I have the following problem:
Let's say I have two polynomials $f,g\in R[X,Y]$ for some domain $R$, and I want to check if $g$ divides $f$. If both $f,g$ were in $R[X]$ then I would do polynomial division and get $f=gh+r$ with $r=0$ or $\deg(r)<\deg(g)$.
I wanted to do the same now. My Professor said it's not that easy because you'd have to choose an order of the monomials and it would get more complicated. I have read something on Wikipedia about that as well. But I don't understand why this is necessary. Why can't I just think of $f,g$ as elements of $R(Y)[X]$ which is an Euclidean domain and by polynomial division I get $f=gh+r$ with $r=0$ or $\deg(r)<\deg(g)$.
Now if $r,h\in R[Y][X]$ and $r=0$ then we have found an equation $f=gh$ in $R[X,Y]$ and $g$ divides $f$.
If $g$ divides $f$ then $f=g\tilde{h}$ for $\tilde{h}\in R[X,Y]$ and since the representations of $f$ is unique we have $h=\tilde{h}$ and $f=hg+r=\tilde{h}g+0$
If $r\neq 0$ then certainly $g\nmid f$ in $R(Y)[X]$ and in particular not in $R[X,Y]$
So where is the problem with that? I assume there is some, but I don't know why. I would be very thankful if you could get me a hint. Thank you for your time!
I think the polynomial part might just be a distraction. If $R$ is a domain, then so is $R[x]$ and $R[x,y]$. Division with remainder does not work well in $R[x]$, but it is not needed.
In any integral domain $R$ to check that $g$ divides $f$ you could say "compute the element $\frac{f}{g} \in K$ where $K$ is the field of fractions of $R$, and if the result is in $R$, then $g$ divides $f$." No mention of degrees or anything.
Your idea to use division with remainder may run into problem. For example given $g(x) = 2x$ and $f(x) = x^2$ in $R[x] = \mathbb{Z}[x]$ then there are no polynomials $h,r \in \mathbb{Z}[x]$ so that $f = gh + r$ and $\deg(r) < \deg(g)=1$.
This is not a big deal, as @user26857 mentions, you just work with the field of fractions of $R$ instead. My suggestion is not to stop at $K[x]$ or $K(x)$, but really just keep going all the way to $K(x,y)$.
I think the problem is a little clearer now: how do you tell if something in the field of fractions of a domain is actually in the domain? how do you tell if something in $K(x,y)$ is in $R[x,y]$? how do you tell if something in $K$ is in $R$?
For example, is $\frac{x^2}{2x} \in \mathbb{Z}[x]$? (no, we can use the unique lowest terms form to decide)
Is $\frac{1+\sqrt{-5}}{1-\sqrt{-5}} \in \mathbb{Z}[\sqrt{-5}]$? No, but it is more challenging to decide, since both $\frac{1+\sqrt{-5}}{1-\sqrt{-5}}$ and $\frac{-2+\sqrt{-5}}{3}$ are in lowest terms (and are equal). We need to use an integral basis here to decide which fractions from $\mathbb{Q}[\sqrt{-5}]$ are in $\mathbb{Z}[\sqrt{-5}]$.
So I think the key thing you need is to have a clear way of deciding whether elements of a field of fractions are within the domain. I think this is the only problem I see with your method: in the very last step, how do you tell if the $K$-coefficients in your $K[x][y]$ polynomials all lie in $R$? If that is not a problem, your method works. If that is difficult (and it could be for weird $R$), then I think your method won't work, but the polynomials won't be the problem.