Let $M$ be a matrix in $ \operatorname{GL}(2, \mathbb{Z})$, are there any books/articles that give a description of $\operatorname{Aut}(\mathbb{Z}^2\rtimes_M\mathbb{Z})$?
According to this paper, the automorphism group of any polycyclic group is isomorphic to a linear group. I am trying to understand how the automorphism group is embedded into the matrix group. I was wondering if anybody has invested this particular example $\operatorname{Aut}(\mathbb{Z}^2\rtimes_M\mathbb{Z})$ and gave a description of this group.
Thank you!
If $M$ is not virtually unipotent (i.e. has trace $\neq 0,\pm 1,\pm 2$), then the normal $\mathbf{Z}^2$ is the nilpotent radical (aka Fitting subgroup) and hence is preserved by automorphisms. Suppose so.
Say an automorphism is positive if the induced automorphism of $\mathbf{Z}$ (obtained after modding out the normal $\mathbf{Z}^2$) is identity. Such an automorphism acts on the normal $\mathbf{Z}$ as some element $M'$ of $\mathrm{GL}_2(\mathbf{Z})$ commuting with $M$. Conversely such $M'$ induces a positive automorphism $(v,n)\mapsto (M'v,n)$.
Hence, letting $A_+$ be the group of positive automorphisms, we have $A_+=C\ltimes A_+^1$, where $C$ is the centralizer of $M$ in $\mathrm{GL}_2(\mathbf{Z})$, and $A_+^1$ consists of those positive automorphisms acting as identity on $\mathbf{Z}^2$. The latter is isomorphic to $\mathbf{Z}^2$ (inside which the image of $\mathrm{Id}_2-M$ consists of inner automorphisms), while $C$ contains the group of inner automorphisms $\langle M\rangle$ with finite index. The group $C$ is itself infinite cyclic.
There are negative automorphisms if and only if $M$ is conjugate to $M^{-1}$ in $\mathrm{GL}_2(\mathbf{Z})$. I don't know if it is always the case (they are indeed conjugate in $\mathrm{GL}_2(\mathbf{Q})$).
The remaining cases: - when $M$ has infinite order, i.e., $M$ is either unipotent nontrivial, or the negative thereof. Up to conjugation it means that $M= \pm u_k$, $u_k=\begin{pmatrix}1 & k\\ 0 & 1\end{pmatrix}$. - when $M$ has finite order (1,2,3,4, or 6).
In the case where 1 is not an eigenvalue, then the derived subgroup has finite index in $\mathbf{Z}^2$, and $\mathbf{Z}^2$ can be obtained as inverse image of subgroup of torsion elements in the quotient. So we can argue similarly as in the previous case, although the description of $C$ and $A_+$ is a bit different.
Remains the unipotent case. If $M$ is identity the automorphism group is $\mathrm{GL}_3(\mathbf{Z})$. Otherwise up to conjugation $M=u_k$ for some $k\ge 1$. For $u_1$, the automorphism group is isomorphic to $\mathrm{GL}_2(\mathbf{Z})\ltimes \mathbf{Z}^2$. I haven't computed for larger $k$, it's probably similar (although this might slightly change the outer automorphism group).