I am trying to solve the following question;
Given a polynomial $f\in \mathbb{C}[x_{1},x_{2},\ldots,x_{n}]$, find the largest subgroup $\Gamma\le GL(\mathbb{C}^{n})$ such that $f\in \mathbb{C}[x_{1},x_{2},\ldots,x_{n}]^{\Gamma}$. Here $f\in \mathbb{C}[x_{1},x_{2},\ldots,x_{n}]^{\Gamma}$ denotes the invariant ring of $\Gamma$, i.e the set of all $\sigma\in GL(\mathbb{C}^{n})$ such that $f\circ\sigma=f$.
I have tried simple cases (quadratic and linear polynomials) by just taking a polynomial say $f=x^{2}+y^{2}$ and a generic matrix $A=\begin{bmatrix}p & q\\r & s\end{bmatrix}$, then computing $$A \begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}ax+by\\cx+dy\end{bmatrix},$$which I then plug into $f$ to get $$(px+qy)^{2}+(rx+sy)^{2}=p^2 x^2+r^2 x^2+2 p q x y+2 r s x y+q^2 y^2+s^2 y^2.$$ Then I equate coefficients to get a system of equations $$\{p^2+r^2-1, 2pq+2rs, q^2+s^2-1\}$$ which I solve using Groebner bases.
This method works for these basic cases, however seems very basic and without much theory behind it. I was wondering if there was a more efficient and/or elegant way to compute these groups given a generic polynomial?
Background wise, I am familiar with all content from the book https://www.springer.com/us/book/9783211774168, which covers classical invariant theory (Hilbert Series, Molien Series, Noethers bound, etc.).
Any help is appreciated.
The object you seek is also called the stabilizer $G_f$ of $f$ under the action of $\operatorname{GL}_n$. When you truly ask for the stabilizer of a generic polynomial: It is quite easy to show that the stabilizer of almost every polynomial is trivial, and if you assume $f$ to be homogeneous of degree $d$ then in the generic case it will only contain those scalar matrices which correspond to $d$-th roots of unity.
Granted, this was a bit cheeky. If you are asking for an algorithm to compute the stabilizer of any given polynomial $f$: This problem is not entirely solved or understood.
The stabilizer of a polynomial will always consist of several connected components all of which are isomorphic (as varieties) and the connected component containing the identity matrix is a subgroup which we call the identity component. If you have an explicit description of your polynomial, you can compute the Lie algebra of the identity component, which some people will accept as a good description for the identity component itself. In particular, this gives a straightforward algorithm to compute the dimension of $G_f$.
However, even in the case where the identity component is trivial, i.e. when the stabilizer is a finite group, I am not aware of any fast algorithm to compute the points of $G$.
The stabilizers of several interesting families of polynomials have been determined over the years, and one successful method has been the following, however it is far from being an algorithm:
Note first that $G_f$ acts on the zero set $Z(f)=\{ z\in\Bbb C^n \mid f(z)=0 \}$ because for $z\in Z(f)$ and $\sigma\in G_f$, you have $f(\sigma(z))=(\sigma^{-1}.f)(z)=f(z)=0.$ Now consider the set $\mathcal L_k$ of all linear subspaces $L\subseteq Z(f)$ with $\dim(L)=k,$ then $G_f$ also acts on $\mathcal L_k$ because $\sigma(L)$ is again a linear space of dimension $k$ and because $G_f$ acts on $Z(f),$ it follows from $L\subseteq Z(f)$ that also $\sigma(L)\subseteq Z(f).$ Now, by studying the sets $\mathcal L_k$ for different values of $k$, it is sometimes possible to determine the group $G_f$ quite explicitly. For example, this technique yields a very nice way to determine the stabilizer of the determinant polynomial.
For a much more simple example of this technique, I will point you to Chapter 9 of my thesis, in particular 9.1. There's a PDF there somewhere:
https://depositonce.tu-berlin.de/handle/11303/6524
(I hope this alright with math.se standards because it is a published document)