Let $G=\langle a,b\mid a^4=b^4=1, bab^{-1}=a^3\rangle$.
Please prove that $Aut(G)$ is generated by the automorphisms $$a\mapsto ab,\hspace{10pt} a\mapsto a^3,\hspace{10pt} a\mapsto ab^2, \hspace{10pt}b\mapsto a^2b \hspace{6pt}\text{and}\hspace{6pt}b\mapsto b^3.$$ and characterize the automorphisms group of $G$.
Explaintion for comment john mangual: Here we define the automorphisms on the the generatos $a$ and $b$. for example one of the automorphisms is $a\mapsto ab$, $b\mapsto b^3$. I think that the generators of $Aut(G)$ are the above form. Only i can not prove it.
Thank you
There is a natural isomorphism of $G$ with a semi-direct product $$\langle a \rangle \rtimes\langle b\rangle\simeq \mathbb{Z}/4\mathbb{Z}\rtimes \mathbb{Z}/4\mathbb{Z}$$ and where the action of b on $\langle a\rangle$ is given by $$\begin{array}{ccl} \langle a \rangle & \to & \langle a \rangle \\ a^i& \mapsto & a^{3i}=a^{-i}.\end{array}$$ Hence, $a^2$ commutes with $b$ and $a$ commutes with $b^2$. You can check that $$Z(G)=\langle a^2,b^2\rangle\simeq \mathbb{Z}/2\mathbb{Z}\times \mathbb{Z}/2\mathbb{Z}.$$ (the inclusion is clear and neither $a$, nor $b$ nor $ab$ belongs to $Z(G)$). Moreover, the commutator subgroup of $G$, that you write $G'$, contains $a^{-1}bab^{-1}=a^2$ and is in fact equal to $$G'=\langle a^2\rangle \simeq \mathbb{Z}/2\mathbb{Z}.$$
Hence, you can check that $H=\{g\in G \mid g^2\in G'\}=\langle a,b^2\rangle$ (it is a maximal subgroup, and $b\notin H$).
Let us show that $\mathrm{Aut}(G)$ is generated by
$$\begin{array}{rcl} \tau_1\colon &G&\to& G\\ &a&\mapsto &ab^2\\ &b&\mapsto &b\end{array},\hspace{1cm} \begin{array}{rcl} \tau_2\colon &G&\to& G\\ &a&\mapsto &a^{-1}\\ &b&\mapsto &b\end{array},\hspace{1cm} \begin{array}{rcl} \tau_3\colon &G&\to& G\\ &a&\mapsto &a\\ &b&\mapsto &ab\end{array},\hspace{1cm} \begin{array}{rcl} \tau_4\colon &G&\to& G\\ &a&\mapsto &a\\ &b&\mapsto &b^{-1}\end{array}$$
Take $\tau\in \mathrm{Aut}(G)$. By definition of the subgroups, $\tau$ preserves $Z(G)$, $G'$ and $H$. Since $G'$ is of order $2$, $\tau(a^2)=a^2$.
Moreover, $\tau(a)$ is an element of $H$ whose square is $a^2$, so $\tau(a)\in \{a,a^{-1},ab^2,a^{-1}b^2\}$. Applying compositions of $\tau_1,\tau_2$ to any automorphism, we can assume that $a=\tau(a)$.
We then look at the action of $\tau$ on $G/Z(G)\simeq \mathbb{Z}/2\mathbb{Z}\times \mathbb{Z}/2\mathbb{Z}.$ It fixes the class of $a$ and then either fixes the class of $b$ or exchanges it with the class of $ab$. Applying $\tau_3$ if needed, we can then assume that the class of $b$ is fixed, which means that $\tau(b)\in \{b,b^{-1},a^2b,a^2b^{-1}\}$. Hence, $\tau\in \langle \tau_3,\tau_4\rangle$.