I have this question on my assignment and I this fact seems trivial to me, but I can not come up with a rigorous proof. I thought to go by contradiction:
Assume such a group $G$ is Abelian -> $\forall f,g \in G:f \circ g=g \circ f $. Then $\forall x_i \in G: f(g(x_i) = g(f(x_i)$. $
It implies that if $\exists x_n,x_m \in G: g(x_i) = x_n, f(x_i) = x_m \rightarrow g(x_m) = f(x_n)$, which is the case only if $g = f^{-1}$, but that is obviously not always the case.
Need some pointer to wrap this up in a formal proof.
If $X$ has at least $3$ elements, then the group of all invertible maps on $X$ contains $S_3$, which is not abelian.
Concretely, let $x_1, x_2, x_3$ be three distinct elements of $X$. Then the map that swaps $x_1$ and $x_2$ does not commute with the map that swaps $x_1$ and $x_3$.