I don't know if this is a trivial question. But because I lack some background I would need advise or a reference. I have an $n$-variate polynomial over $\mathbb Q$, say $f$, and I am interested in finding a maximal permutation subgroup $G$ of $S_n$ such that $f$ is $G$-invariant. How do I find this?
I could of course find $G$ by method of exhaustion (just look at all subgroups of $S_n$), but I feel a more efficient techniques in invariant-theory would help me find this $G$. I am actually more interested in the order of $G$ or some (non-trivial) lower and upper bound depending on $f$ (some kind of a formula depending on the coefficients in $f$). I would also be interested in any sort of reference that could help me with this. People seem to be more interested in the reverse question, i.e. given $G$ find all polynomials that are $G$-invariant.
Edit: I just realized that there is a partial answer to this question.
Some basic observations. Let $f \in \mathbb{Q}[x_1, \ldots, x_n]$. Then the subgroup $G$ you are looking for is the stabilizer of $f$ in $S_n$, i.e.
$$G = \operatorname{stab}_{S_n}(f) = \{\sigma \in S_n : f^{\sigma} = f\}$$
So how do you find this subgroup? First you could write $f = f_0 + f_1 + \cdots + f_d$, where each $f_i$ is a homogeneous polynomial of degree $i$. Then $G$ is equal to the intersection $$\cap_{i = 0}^d \operatorname{stab}_{S_n}(f_i)$$
so we could just focus on the case where $f$ is homogeneous, of degree $d$ say.
Since the action of $S_n$ preserves the coefficients, you can reduce to the case where all the coefficients are equal, so without loss of generality $$f = \sum_{j_1 + \cdots + j_n = d} \varepsilon_{j_1, \ldots, j_n}x_1^{j_1} \cdots x_n^{j_n}$$
where $\varepsilon_{j_1, \ldots, j_n} \in \{0,1\}$ (almost all zero).
Now $S_n$ acts naturally on $\mathbb{N}^n$ the set of $n$-tuples with entries in non-negative integers, and in this case $G$ is the stabilizer of the subset $$\Gamma = \{ (j_1, \ldots, j_n) : j_1 + \cdots + j_n = d \text{ and } \varepsilon_{j_1, \ldots, j_n} = 1 \}$$ of $\mathbb{N}^n$.
So the problem of determining the stabilizer of a polynomial is equivalent to the problem of determining $\operatorname{stab}_{S_n}(\Gamma_1)$, $\ldots$, $\operatorname{stab}_{S_n}(\Gamma_k)$ for some $\Gamma_1, \ldots, \Gamma_k \subseteq \mathbb{N}^n$, and computing the intersection $$\operatorname{stab}_{S_n}(\Gamma_1) \cap \cdots \cap \operatorname{stab}_{S_n}(\Gamma_k).$$
There are probably further things one could say, but this seems like a pretty general problem.