Let $H \leqslant G$ a subgroup of a group $G$ with following condition:
$H = \{ g \in G \vert g^{-1} = g \}$
Now I should show, that this is a subgroup of $G$. So far so easy. If I apply this to the permutation group $S_3$, it won't work anymore. E.g. let the two maps
$p_1 = \left(\begin{array}{3,2} 1 & 2 & 3 \\ 2 & 1 & 3\\ \end{array} \right)$ and $p_2 = \left(\begin{array}{3,2} 1 & 2 & 3 \\ 1 & 3 & 2\\ \end{array} \right)$
Both maps are the inverse element of itself, thus the condition would be true. But if we connect those maps, it will result following map:
$p_1 \circ p_2 = \left( \begin{array}{3,2} 1 & 2 & 3 \\ 2 & 3 & 1\\ \end{array} \right)$
This map is not inverse to itself, therefore it is not in $H$. This implies that the group is not closed and not a subgroup by definition. How is this possible?
If we assume that $H$ is a subgroup and take two arbitrary elements $g, h\in H$, then we get $$ (gh)(gh) = e\\ ghghh = h\\ ghg = h\\ ghgg = hg\\ gh = hg $$ which means that $g$ and $h$ commutes.
It turns out that this is exactly what decides whether $H$ is a subgroup or not: whether all elements of $H$ commute with all other elements of $H$. (We clearly have $e\in H$, and by definition of $H$ we also have $h\in H \implies h^{-1} \in H$, so the only thing that isn't automatically fulfilled is closure under multiplication.) This is trivially satisfied if $G$ is abelian, and it is not satisfied for $G = S_3$, as you have noted yourself.