Let $R$ be a UFD and $f,g \in R[x]$ not both zero. Prove that
$$\gcd(f,g) \in R[x] \backslash R \iff res(f,g)=0$$
The hint is to consider $R$ as embedded in its quotient field and rewrite the vanishing of the resultant as a property of a (normalized) GCD.
For $\implies$ we have a $\gcd$ that is non-constant (?). Now because $\gcd(f,g)$ divides the resultant (?) I guess I somehow need to prove that the gcd can't divide the resultant unless it is $0$?
I don't know how to prove the opposite direction. Thanks for any help!
The resultant vanishes iff there are $p,q\in R[x]$, not both zero, with $\deg(p)<\deg(g)$ and $\deg(q)<\deg(f)$ such that $pf+qg=0$. The proof of this should be rather clear from the usual definition of the resultant via the Sylvester matrix.
Suppose that $h=\operatorname{gcd}(f,g)\in R[x]\setminus R$. Then $f=ah$ and $g=bh$ for $a,b\in R[x]$ and since $\deg(h)>0$, we have $\deg(a)<\deg(f)$ and $\deg(b)<\deg(g)$. Finally, $bf-ag=0$, and therefore the resultant vanishes.
Conversely, if the resultant vanishes, there are $p,q\in R[x]$ such that $\deg(p)<\deg(g)$, $\deg(q)<\deg(f)$ and $pf+qg=0$. Supposing that $\operatorname{gcd}(f,g)\in R$, one must have $f|q$ since $-pf=qg$ so any prime factor of $f$ divides $qg$ but it does not divide $g$. However, $\deg(q)<\deg(f)$ and so by contradiction $\operatorname{gcd}(f,g)\not \in R$.
It may be helpful to note that if $R$ is a domain, then $R[x]^\times=R^\times$. Therefore, when $R$ is a UFD $\operatorname{gcd}(f,g)\in R$ implies that $\operatorname{gcd}(f,g)$ is a unit (given $f,g\ne 0$), and thus not a prime/irreducible, i.e. $f,g$ are coprime.