Let $R$ be a finite type integral domain over field $k$, $S$ be a subring of $R$, such that $[f.f(R)\colon f.f(S)]=d$, does there exist a group $G$ with $|G|=d$ such that $S=R^G$? (Here $f.f(R)$ means the fraction field of $R$)
(I am confused by [ACGH] Geometry of algebraic curves Vol II, P.262 Line 14-16. How shall I understand the claim in [ACGH]?
If $S=R^G$ and both $R,S$ are smooth, and the extension is not etale, can we show $\mathrm{Spec}(R)\to\mathrm{Spec}(S)$ has to be a cyclic cover over smooth divisor? I think the following is a counterexample.)
(Original question: Let $k[x,y]$ be the polynomial ring over $k$ with two variables. Does there exist an action of some finite group $G$ on $k[x,y]$ such that $k[x,y]^G=k[x^2,y^2]$? The naive $x\to -x,y\to -y$ has a larger invariant subring $k[x^2,xy,y^2]$.)
No. There need not be a group $G$ such that $S=R^G$.
One obstruction is what is mentioned by Mohan in comments. This is already an obstruction if $R,S$ are fields (say $R$ is a finite field extension of $k$ and $S$ is a subfield). To have $S=R^G$, it must be that $f.f(R)/f.f(S)$ is a Galois extension of fields. So we may take any non-Galois field extension to get a counterexample. For example, let $R = \mathbb{Q}(\sqrt[3]{2})$ and let $S= \mathbb{Q}$. (Here, $k=\mathbb{Q}$ as well.) In this situation, $R$ does not have any nontrivial automorphisms; thus taking an invariant subring cannot yield a smaller ring. More generally, if $f.f(R)/f.f(S)$ is not Galois, then $R$ will not have "enough automorphisms (that fix $S$)" to get an invariant ring small enough to be $S$.
A second obstruction comes from the fact that your setting does not constrain how far $R$ and $S$ are from being fields. If $R$ has "more fractions in it" than $S$ has, then taking invariants will not be able to get rid of these fractions. More precisely, if we had $S=R^G$, this would imply that $R$ is integral over $S$. Thus if $R$ contains elements that are not integral over $S$, we can rule out $S = R^G$. For example, consider $k = \mathbb{Q}$, $R = k[x,x^{-1}]$, and $S = k[x]$. Here, we have $f.f(R) = f.f(S) = k(x)$, so $d=1$. Since $S$ is strictly smaller than $R$, there is no way a group of order $1$ acting on $R$ is going to have $S$ as an invariant ring. But more generally, no finite group can have $S$ as an invariant ring, since $x^{-1}\in R$ is not integral over $S = k[x]$.