This problem is from the book "Introduction to Group Theory" by Derek J.S Robinson. So, the notations are according to the book.
Prove that the only 3-transitive permutation groups of degree $6$ are $S_{6}$, $A_{6}$, and $L(5)$.
First I elaborate a little bit on $L(5)$. Let $F_{5}$ be the finite field of order 5. Consider $F=F_{5}\cup \{\ \infty \}\ $. We call $\alpha:F\to F$ defined by $\alpha(x)=\frac{ax+b}{cx+d}$, where $ad-bc\neq 0$ to be a fractional linear transformation. Then $L(5)$ is the set of all fractional linear transformations. Clearly $L(5)$ is a group under composition of maps. We have $|L(5)|=120$ and $L(5)$ acts on $F$ by evalutation. We have $|F|=6$. It can be shown that kernel of this action is trivial and it is sharply $3-$ transitive. So $L(5)$ is a 3-transitive permutation group of degree $6$.
Now I write what I have tried: Clearly as mentioned $S_{6}, A_{6}, L(5)$ are 3-transitive permutation groups of degree 6. Now, if $G$ is a 3-transitive group of degree 6 , we know that $120||G|$. So $|G|=120, 240, 360, 720$. If $|G|=720$ or $360$, then cleary $S_{6}$ and $A_{6}$ are only choices. Now at this point I am stuck. I have to contradict the fact $|G|=240$. And then I have to prove that if $|G|=120$, then $G\cong L(5)$. But I don't have a clue about how to do it!
Thanks in advance for any kind of help!
To rule out $|G|=240$, observe that $|S_6:G|=3$, so $S_6$ acts transitively on the three-element set of left cosets of $G$. Then $S_6$ will have a normal subgroup of index $3$ or $6$. This would contradict the simplicity of $A_6$.