This is part of an exercise in Abstract Algebra: Dummit & Foote showing that$|\mathrm A\mathrm u\mathrm t(Sym(6)):\mathrm I\mathrm n\mathrm n(Sym(6))|\le2$. (Sec. 4.4, Ex. 19)
In a previous exercise, I've shown that the automorphism group of a group $G$ permutes the conjugacy classes of $G$ in a way that for each $\sigma\in \mathrm A\mathrm u\mathrm t(G)$ and each conjugacy class $\mathcal K$, the set $\sigma (\mathcal K)$ is also a conjugacy class of $G$. And $|\sigma(\mathcal K)|=|\mathcal K|$.
I want to:
deduce that $\mathrm A\mathrm u\mathrm t(Sym(6))$ has a subgroup of index at most $2$ which sends transpositions to transpositions.
I already know why the choice of $6$ is special. Let $\mathcal K$ be the conjugacy class of transpositions in $Sym(6)$. It contains $15$ such 2-cycles. Let $\mathcal K'_1$ be the conjugacy class containing all the products of two disjoint transpositions. There are $45$ of them. Let $\mathcal K'_2$ be that containing all the products of three disjoint transpositions. There are $15$ of them. We have $|\mathcal K|=|\mathcal K'_2|$. i.e. The conjugacy class of transpositions in $Sym(6)$ has order equal to the conjugacy class containing all the products of a particular no. of disjoint transpositions. This is the case for $Sym(6)$ only.
This is what I found from http://math.stanford.edu/~mkahle/hw8_solns.pdf:
Let $H$ be the subgroup of $\mathrm A\mathrm u\mathrm t(Sym(6))$ consisting of automorphisms that takes transpositions to transpositions. Note that $H$ is nonempty, since it contains the identity element... We have shown that, given any automorphism $\sigma$ of $Sym(6)$, the automorphism $\sigma^2$ takes transpositions to transpositions. That is, $\sigma^2\in H$. This means $H$ has index at most $2$ in $\mathrm A\mathrm u\mathrm t(Sym(6))$.
but I noticed that in general, even if for all $x\in G$, $x^2\in H$, it may not be true that $H$ is a subgroup of index at most $2$: Take $G=Dih(4)$, $H=\{x^2\}=\{1,r^2\}$ is a subgroup of index $4$. So, the proof is false, right?
By the way, I have no idea how to begin writing my proof. I only know that as $\sigma\in \mathrm A\mathrm u\mathrm t(Sym(6))$, $\sigma$ permutes conjugacy classes and preserves order of elements. So either $\sigma(\mathcal K)=\mathcal K$ or $\sigma(\mathcal K)=\mathcal K'_2$... This is where I get lost.
Anyway, many thanks to those who read the whole thing!!!! :D
The argument is indeed not correct. However, it can be easily fixed. Note that for automorphisms $\sigma_1$ and $\sigma_2$ with $\sigma_i(\mathcal{K}) = \mathcal{K}_2'$ you have $\sigma_1^{-1} \sigma_2(\mathcal{K}) = \mathcal{K}$, so $\sigma_1^{-1}\sigma_2 \in H$. This shows that $H$ has at most one non-trivial coset in $Aut(Sym(6))$, thus its index is at most two.
This is essentially an example of the orbit-stabilizer theorem. $H$ is the stabilizer of $\mathcal{K}$ on the action of $Aut(Sym(6))$ and the orbit of $\mathcal{K}$ has length $1$ or $2$ (it is $\{\mathcal{K}\}$ or $\{\mathcal{K}, \mathcal{K}_2'\}$), thus $H$ has index $1$ or $2$.