I'm working on the following problem:
Let $G$ be the group $\langle a,b \mid aba^{-1} = b^2\rangle$.
a) Show that the map $\phi: G \longrightarrow GL_n(\Bbb Q)$ defined by $\phi(a) =\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} = A$ and $\phi(b) = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} = B$ is a group homomorphism.
b) Prove that $A^nBA^{-n} = B^{2^n}$ for all positive integers $n$.
c) Prove that $G$ is infinite.
For part a), I know that such a group homomorphism $\phi: G \longrightarrow GL_n(\Bbb Q)$ would have the property that $\phi(ab) = \phi(a) \cdot \phi(b)$. But although we know $\phi(a)$ and $\phi(b)$ individually, how do we know what $\phi(ab)$ is equal to ? I can get from the relations that $ab = b^2a^{-1}$, but I'm not sure how to proceed from there. Does $\phi(ab) =\phi(b^2a^{-1}$) buy any useful relation to tell us what $\phi(ab)$ must equal?
I was able to prove part (b). We proceed by induction. The base case $(n=1)$ checks out -- we can verify through matrix multiplication that $ABA^{-1} = B^2$. Note that this also means that $AB^jA^{-1} = (B^2)^j$, where $j$ is a positive integer. Now, assume $A^{n-1}BA^{-n+1} = B^{2^{n-1}}$. Then $A^nBA^{-n} = AB^{2^{n-1}}A^{-1} = (B^2)^{2^{n-1}} = B^{2^n}$, and the result follows.
For part c), my idea so far was to show that $ab$ has infinite order. Using $ab = b^2a^{-1}$, I planned on showing $a^n \neq e$ for any $n \in \{0,1,2,...\}$, but I'm not sure how to proceed.
Thanks!
(a) To show that the given map is a homomorphism (in the category of groups), we have to show that relations go to relations. The given relation $aba^{-1}=b^2$ in the presentation of $G$ goes to the (possibly invalid) relation $ABA^{-1}=B^2$, and we compute for this: $$ \begin{aligned} AB&= \begin{bmatrix} 2&\\&1 \end{bmatrix} \begin{bmatrix} 1&1\\&1 \end{bmatrix} = \begin{bmatrix} 2&2\\&1 \end{bmatrix} \\ B^2A &= \begin{bmatrix} 1&1\\&1 \end{bmatrix}^2 \begin{bmatrix} 2&\\&1 \end{bmatrix} = \begin{bmatrix} 1&2\\&1 \end{bmatrix} \begin{bmatrix} 2&\\&1 \end{bmatrix} = \begin{bmatrix} 2&2\\&1 \end{bmatrix} \ , \end{aligned} $$ so we have indeed $AB=B^2A$, equivalently $ABA^{-1}=B^2$.
(b) Induction. The relation in (a) is the beginning. Fix an $n\ge 1$. Assume the relation $A^nBA^{-n}=B^{2^n}$ is true for this $n$. Then: $$ \begin{aligned} A^{n+1}BA^{-(n+1)} &= A\ A^nBA^{-n}\ A^{-1} \\ &=A\ B^{2^n}\ A^{-1} \text{ by assumption} \\ &=(ABA^{-1})^{2n} \\ &=(B^2)^{2^n} \\ &=B^{2^{n+1}} \ . \end{aligned} $$ This shows (b) inductively.
(c) The composition of the representation $\phi$ with the determinant morphism gives a map $G\to \Bbb Q^{\times}$, the image of this morphism is the infinite group generated by $2$, so $G$ is infinite. (Or observe that either $A$, or $B$ has infinite order, by letting the matrices act as usual on the rational or real plane, or on the complex plane using the homographic action $\begin{bmatrix}a&b\\c&d\end{bmatrix}\cdot z = (az+b)/(cz+d)$. By this last actions, $B$ corresponds to the translation by one.)