This is concerning part of Exercise 7.1.4 of Robinson's, "A Course in the Theory of Groups (Second Edition)". According to this search of "similarity type" in the group-theory tag, it is new to MSE.
The Details:
(This can be skipped.)
Arguments are written on the left of functions.
A permutation group is, given a nonempty set $X$, a subgroup $G$ of $S_X$; its degree is the cardinality of $X$.
A permutation group $G$ on $X$ is transitive if, for any $x,y\in X$, there is a permutation $\pi\in G$ such that $x\pi=y$
A similarity from permutation groups $G$ and $H$ on sets $X$ and $Y$, respectively, is a pair $(\alpha, \beta)$ consisting of an isomorphism $\alpha:G\to H$ and a bijection $\beta:X\to Y$ such that
$$\pi\beta=\beta\pi^{\alpha}$$
for all $\pi\in G$.
A similarity type is an equivalence class of similarities.
The Question:
Here is Exercise 7.1.4:
List all similarity types of transitive permutation groups of degree less than or equal to five.
However, I am interested in what the answer would look like, not a full solution. The reason why, I hope, will become clear from my thoughts below.
Thoughts:
According to GroupNames, $S_5$ has $156$ subgroups; I doubt I have to sieve through them all, looking for transitive groups.
But I don't know how to do this exercise without brute force.
Surely there's a more sophisticated way . . .
I don't suppose it'd be difficult to put this in GAP. I haven't had chance to over the last couple of weeks, due to other commitments.
I want to ask this question sooner rather than later. I'm starting to forget the preceding material of the book.
The kind of answer I'm looking for is a description of what a solution to the exercise would look like, with an emphasis on techniques beyond brute force.
Please help :)
If the group is transitive of degree $5$ it must contain a $5$-cycle, and that must generate a $5$-Sylow subgroup, call it $S$. If $S\triangleleft G$, then $G$ is (up to similarity) a subgroup of $N_{S_5}(\langle(1,2,3,4,5)\rangle)$, which is a semidirect product of $S$ with $C_4$. (There are 3 possibilities, orders 5,10, 20. Otherwise, there is more than one $5$-Sylow subgroup, which implies that the order of $G$ is at least $5\cdot 6=30$. But $S_5$ has no sugroup of index $4$ (would give homom. into $S_4$, and normal subgroup that does not exist. So then $|S|\in\{60,120\}$ with $A_5$ and $S_5$ as the only possibilities.