Consider the group
$$G = \langle a,b\mid a^5=b^{11}=1,\ aba^{-1}=b^3\rangle.$$
I wish to understand $\operatorname{Aut}(G)$, the automorphism group of this presentation. I could not find a systematic way to solve this. Using Magma, I found that this should be a group of order $110$; specifically, it is the group generated by the two maps $\phi^1$ and $\phi^2$, where these are given by $\phi^1(a) = a$, $\phi^1(b) = b^6$ and $\phi^2(a) = ab$, $\phi^2(b) = b$ respectively. Hence I believe the group to be $\mathbb{Z}_2 \rtimes \mathbb{Z}_{55}$ but how can this be found.
My solution: ${\rm Aut}(G)\cong \Bbb Z/11\Bbb Z \rtimes \Bbb Z/10\Bbb Z$.
Start by studying the group $G$: Since $aba^{-1}=b^3$, then also $a^{-1}ba=b^4$ and so every element in $G$ has the form $a^hb^k$ for $0\le h\le 4$ and $0\le k\le 10$. In particular, $G$ is a non abelian group of order $55$ and thus it is $$G\cong \Bbb Z/11\Bbb Z\rtimes \Bbb Z/5\Bbb Z.$$ $G$ has only one $11$-Sylow, given by $\langle b\rangle$ and all the other elements have order $1$ or $5$.
Calculating the order of ${\rm Aut}(G)$: Let $\phi\in{\rm Aut}(G)$, because of order reasons $\phi(a)=a^nb^m$ and $\phi(b)= b^k$ for such $1\le n\le 4$, $0\le m\le 10$ and $1\le k\le 10$. Imposing the condition $\phi(aba^{-1})=\phi(b^3)$, we obtain the condition $a^nb^ka^{-n}=b^{3k}$, that becomes $b^{3^nk}=b^{3k}$, i.e. $n= 1 \mod 5$ and then $n=1$. So $\phi(a)=ab^m$ and $\phi(b)=b^k$ for $0\le m\le 10$ and $1\le k\le 10$. We conclude that $|{\rm Aut}(G)|=11\cdot 10=110$.
Studying the structure of ${\rm Aut}(G)$: Since ${\rm Aut}(G)$ is a group of order $2\cdot 5\cdot 11$ you can conclude suddenly that there exists a normal subgroup of order $55$ (search for “$pqr$-groups”) and then thanks to Cauchy’s Theorem ${\rm Aut}(G)\cong (\Bbb Z/11\Bbb Z\rtimes \Bbb Z/5\Bbb Z) \rtimes \Bbb Z/2\Bbb Z $.
Another way, with some counts, is the following.
The $j$-th iteration of $\phi$ gives $\phi^j(b)=b^{k^j}$ and then if we want an automorphism of order multiple of $10$ we can ask that the multiplicative order of $k\mod 11$ is $10$, for example we can choose $k=2$. So, we fix the automorphism $\Phi(a)=a$ and $\Phi(b)=b^2$. Similarly, we search for an automorphism of order multiple of $11$: this time we may ask $k=m=1$ and we see that $\phi^j(a)=ab^j$, that in fact has order $11$. So, we fix the automorphism $\Psi(a)=ab$ and $\Psi(b)=b$. We finally observe that $\Phi\circ\Psi\circ \Phi^{-1}=\Psi^2$ and so ${\rm Aut}(G)$ is a non abelian group of order $110$, product of $\langle \Psi\rangle$ and $\langle\Phi\rangle$ with the right orders and such that the second subgroup stays in the normalizator of the first. We conclude that $${\rm Aut}(G)= \langle \Psi\rangle\rtimes\langle\Phi\rangle\cong \Bbb Z/11\Bbb Z \rtimes \Bbb Z/10\Bbb Z.$$