Large subgroups of Symmetric Group

Question

Let $n \geq 5$. It is easy to prove that if $H \leqslant S_n$ has index $\leq n$, then $H$ is $A_n$ or $S_{n-1}$.

So my question: can this be improved to $n^2$? I mean, if $[S_n:H] \leq n^2$, is it true that $H$ is one among $A_n$, $S_{n-1}$, $A_{n-1}$, $S_{n-2}$, $S_{n-2} \times S_2$?

Answer 1

As noted by others what you are asking for is not true in general. You also missed two infinite families which lie in between $A_{n-2}$ and $S_{n-2} \times S_2$. One of them is $A_{n-2} \times S_2$, and the other one is a copy of $S_{n-2}$ realized as $\{(\sigma, \operatorname{sgn}(\sigma)) : \sigma \in S_{n-2}\}$ in $S_{n-2} \times S_2$.

But besides this there are only finitely many other examples.

See Theorem 5.2B in Permutation Groups by Dixon and Mortimer, which states the following:

Let $S = \operatorname{Sym}(\Omega)$ and $A = \operatorname{Alt}(\Omega)$, where $n = |\Omega| \geq 5$. Suppose that $G \leq S$ has index $[S:G] < \binom{n}{r}$ for some $1 \leq r \leq n/2$. Then one of the following holds:

1. There exists $\Delta \subseteq \Omega$ with $|\Delta| < r$ such that $A_{(\Delta)} \leq G \leq S_{\{\Delta\}}$.
2. $n = 2m$ is even, $G$ is imprimitive with two blocks of size $m$, and $[S:G] = \frac{1}{2}\binom{n}{m}$.
3. One of six exceptional cases hold, with $(n,r,[S:G])$ equal to one of following:
   • $(6,3,15)$, $(5,2,6)$, $(6,2,6)$, $(6,2,12)$, $(7,3,30)$, $(8,3,30)$.

(For details about the exceptional cases, see Dixon and Mortimer's book.)

Apply this result with $r = 3$, and you will find that except for a finite number of examples (which can be determined case-by-case), $[S_n:H] \leq n^2$ implies that up to conjugacy in $S_n$, the subgroup $H$ is one of the following: $S_n$, $A_n$, $S_{n-1}$, $A_{n-1}$, $S_{n-2}$, $S_{n-2} \times S_2$, $A_{n-2} \times S_2$, the other copy of $S_{n-2}$ in $S_{n-2} \times S_2$ described above.

With enough patience, I guess you could use the same approach to classify $H < S_n$ with $[S_n : H] \leq n^3$.

Answer 2

The group $S_{2n}$ has a subgroup $S_n\wr S_2$, induced by partitioning $\{1,\ldots,2n\}$ into two sets of $n$ elements. This is a subgroup of order $2(n!)^2$. When $n=3$, you get a subgroup of order $72$ inside the group of order $6!=720$, that is, index $10\lt 6^2$. This subgroup is the semidirect product to $(S_3\times S_3)\rtimes S_2$, which I think is not on your list.