This is an exercise from Ideals, Varieties and Algorithms by Cox et al.
Denote the finite matrix group $\{\pm I_2\}\subset GL(2,k)$ by $C_2$. It is known that the invariant polynomials under $C_2$ can be written as polynomials of $x^2,xy,y^2$. Hence we can denote this space by $k[x^2,xy,y^2]$.
The exercise asks to find the invariant polynomial space under $C_6$, a cyclic finite matrix group generated by $$A=\begin{pmatrix}0&1\\-1&1\end{pmatrix}$$ using the fact that this space is in $k[x^2,xy,y^2]$ which I have proved, and the fact $$k[x_1,\dots,x_n]^G=k[R_G(x^{\beta}):|\beta|\leq |G|)]$$ where $R_G(f)$ is the Reynold operator acting on $f$ under the group $G$.
So in this case I calculated the Reynolds operator acting on $x,y,x^2,xy,y^2,\dots,x^6,x^5y,x^4y^2,x^3y^3,x^2y^4,xy^5,y^6$.
I found out the following polynomials (I omitted the ones that wouldn't give any new generators): $$x^2-xy+y^2\\ (x-y)^6+x^6+y^6\\ x^5y+ x(x - y)^5 - y^5(x - y)\\ x^4y^2 + x^2(x - y)^4 + y^4(x - y)^2\\ x^3y^3 + x^3(x - y)^3 - y^3(x - y)^3\\ x^2y^4 + x^4(x - y)^2 + y^2(x - y)^4\\ xy^5 + x^5(x - y) - y(x - y)^5\\ (x - y)^6 + x^6 + y^6$$
Since these don't generate $x^2$, this space is included in $k[x^2,xy,y^2]$, but not equal to it. So I have to find some generators. I tried combinations of $x^2,xy,y^2$, for example, $x^2-xy+y^2, x^2-xy$, or $x^2-xy,y^2-xy$, and many others, nothing works.
I hope I made myself clear about this problem. Can anyone suggest a set of generators? Thank you very much!
For simplicity, assume that $k$ is algebraically closed of characteristic zero. If $C_n\subset Gl_2(k)$, a cyclic group of order $n$ and $\sigma$ a generator, one can diagonalize it and thus assume $\sigma$ has diagonal entries $\omega^i,\omega^j$ where $\omega$ is a primitive $n^{\rm{th}}$ root of unity and $i,j,n$ have no common factors. Then the action is given by $\sigma(x)=\omega^ix, \sigma(y)=\omega^jy$ and thus the invariants are given by $x^ay^b$ where $ai+bj$ is divisible by $n$.