Given $\phi: G\rightarrow Aut(G), g\mapsto g^*$
and $g^*: G\rightarrow G, x\mapsto gxg^{-1}$
where $g,g^{-1},x\in G$
I need to prove that $\phi$ is a homomorphism ($\phi(gh)=\phi(g)\phi(h)$).
So, I started like this:
$\phi(gh)=ghx(gh)^{-1}=ghxg^{-1}h^{-1}$ <-here's where I'm stuck.
I considered this step: $=gh(gg^{-1})xg^{-1}h^{-1}=(ghg^{-1})(gxg^{-1})h^{-1}=g^*g^*h^{-1}$,
but somehow that looks wrong.
How do I go from here?
I see two misunderstandings here:
As addressed in the comments, for $g, h$ in any group $G$ we have $(gh)^{-1} = h^{-1}g^{-1}$, not $g^{-1}h^{-1}$.
The operation in $\operatorname{Aut}(G)$ is function composition.
Therefore for every $g,h,x \in G$ we have $$ \phi(gh)(x) = (gh)x(gh)^{-1} = g(hxh^{-1})g^{-1} = g\big(h^*(x)\big)g^{-1} = g^*\big(h^*(x)\big) = (g^* \circ h^*)(x) $$ or, in other words, $\phi(gh) = \phi(g) \circ \phi(h)$.