Dummit-Foote Abstract Algebra Chap.2 sec 2.2 problem 12 (e)
Exhibit all permutation in $S_4$ that stabilize the element $x_1x_2+x_3x_4$ and prove that they formed a subgroup isomorphic to the dihedral group of order 8.
I can solve this ${(1),(12), (34), (12)(34), (1324),(13)(24),(1423), (14)(23)} $
But I can't understand how to handle a general case, for $S_n $. Please help. Thanks for reading.
I believe that your question has no answer, although there are answers to related questions.
The fundamental reason behind this is that you're effectively asking "what happens when permutations act on an arbitrary polynomial?" Well, you can't specify neither the permutations nor the polynomial - this is just intractable (or the answer is simply "anything can happen"). It's like asking "what happens when I evaluate $P(x)$ for some $x$ and some $P(x)$?" Well, anything.
However, if you're interested in how any permutation acts on a specific polynomial (or class of polynomials, chosen with some insight into a particular idea), then maybe we can get somewhere. Conversely, if you want to ask how a specific permutation (or class of permutations, chosen with some insight into a particular idea) acts on any polynomial, then also perhaps we can get somewhere.
So maybe you'd like to ask "Which polynomials are stabilized by $H \leq S_n$?" Or maybe you'd like to ask "Given a specific $P(x_1,\dots,x_n)$, is there a better way to find the subgroup of $S_n$ that stabilizes it than simply trying out every permutation?"
Otherwise, generally: key ideas would be looking at the representation theory of $S_n$, characters of an irreducible representation of $S_n$, and if you were interested in subgroups of $S_n$, then you would be looking at restricted representations. Sagan's book is a good introduction to the theory.
On the other hand, you might want to ask whether your polynomial or class of polynomials is generated from some other set of polynomials whose symmetry properties are well-studied. Useful polynomials to think about are the Schur polynomials and perhaps the idea of a Gröbner basis could help you find a generating set of polynomials.
Certainly other ideas exist, but this is where I'd start.