Hope you are well.
Im posting the following as a favor on behalf of a friend who is not a native speaker, she just wants to confirm her answer is indeed correct.
Q: show that the set of all automorphism functions of G is a group under composition.
A: (h ∘ g)∘ f = h ∘ (g∘ f)
(h ∘ g)(a)∘ f(a) = h( g(a))∘ f(a) =h [ f(a)g(a) ]
h(a)∘(g ∘ f)(a) =h(a) ∘ [ g(f(a))]= h [f(a)g(a)]
Thanks in advance!
Note that $\operatorname{Aut}(G)\subseteq \operatorname{Sym}(G)$, and then $\operatorname{Aut}(G)$ is a group if and only if $\operatorname{Aut}(G)\le\operatorname{Sym}(G)$. Therefore we have to prove that $\forall \sigma,\tau\in\operatorname{Aut}(G)$, both $\sigma\tau\in\operatorname{Aut}(G)$ and $\sigma^{-1}\in\operatorname{Aut}(G)$, namely that both $\sigma\tau$ and $\sigma^{-1}$ are operation-preserving:
\begin{alignat}{1} (\sigma\tau)(gh) &= \sigma(\tau(gh)) \\ &= \sigma(\tau(g)\tau(h)) \\ &= \sigma(\tau(g))\sigma(\tau(h)) \\ &= (\sigma\tau)(g)(\sigma\tau)(h) \\ \tag 1 \end{alignat}
and:
\begin{alignat}{1} gh=gh &\Rightarrow \sigma(\sigma^{-1}(gh))=\sigma(\sigma^{-1}(g))\sigma(\sigma^{-1}(h)) \\ &\Rightarrow \sigma(\sigma^{-1}(gh))=\sigma(\sigma^{-1}(g)\sigma^{-1}(h)) \\ &\Rightarrow \sigma^{-1}(gh)=\sigma^{-1}(g)\sigma^{-1}(h) \\ \tag 2 \end{alignat}