I gave myself the exercise of determining $$\langle a,b \mid aba^{-1}=b^{-1}\rangle$$ which appears in nature as the fundamental group for the klein bottle (deduced from van kampen's theorem.) One way to understand this is as the semidirect product $\mathbb Z \ltimes_{\phi} \mathbb Z$
where conjugation by an element in $\mathbb Z$ acts on $\mathbb Z$ by $\phi_n(m):=(-1)^n(m)$.
I received an assignment to prove some basic properties of $\langle a,b \mid bab^{-1}=a^2\rangle$ including a faithful matrix representation. However, upon doing some calculations I realized what I was doingg was the same as before: studying the action of conjugation by an element.
On one hand, it is clear from $aba^{-1}=b^2$ that the action of some $n \in \mathbb N$ should be something like multiplication by $2$. However, in trying to determine $a^{-1}b a$ I needed some division by $2$, which led me to believe that the action I should be considering is more like
$$\mathbb Z \to_{\phi} \mathrm{Aut}(\mathbb Z[\frac{1}{2}])$$
where $\mathbb Z[\frac{1}{2}]$ is the additive group of $2$-adics. The action seems to be $\phi_n(m):=2^{n}(m)$.
Question 0: Does everything so far make sense?
Furthermore, in my problem, I was given two generators that map to $GL(2,\mathbb R)$.
Question 1: would it be more natural to study the action on $\mathbb R^4$? In particular, this would not be an irreducible representation, but we could have something like
$$\begin{pmatrix}2 & 0 &2 & 0\\ 0 & 1&0&2\\ 0 & 0 & 1 & 1\\ 0 & 0& 0 & 1 \end{pmatrix}$$
where $\mathbb Z$ embeds into the upper left quadrant, and $\mathbb Z[1/2]$ into the bottom right?
Question 2.1: Looking at this matrix got me wondering: how do elements of $\mathbb Z[1/2]$ look like as elements in the presentation $<a,b>$, I can't actually identify this with a subgroup, and how can I prove normality and that there actually is a subgroup that looks like $\mathbb Z[1/2]$?
I would try and show that the homomorphism from your group $G$ to $GL_2(\Bbb{Q})$ given by $$ b\mapsto \pmatrix{2&0\cr0&1\cr},\qquad a\mapsto \pmatrix{1&1\cr0&1\cr} $$ is injective. The image of that homomorphism is the group $$ \Gamma=\left\{\pmatrix{x&y\cr0&1\cr}\bigg\vert\ x=2^n, y=m/2^k, m,n,k\in\Bbb{Z}\right\}. $$ The subgroup $N$ consisting of the elements of $\Gamma$ with $x=1$ is clearly normal. It is the image of the group generated by conjugates of $a$. A mildly unusual thing is that $a$ is conjugate to its "square root". By that I mean the element $$ a^{1/2}:=b^{-1}ab. $$ After all, $$ a^{1/2}a^{1/2}=b^{-1}abb^{-1}ab=b^{-1}a^2b=a. $$ We also have $$a^{1/4}:=b^{-2}ab^2,$$ as $$ a^{1/4}a^{1/4}=b^{-2}a^2b^2=b^{-1}(b^{-1}a^2b)b=b^{-1}ab=a^{1/2}. $$ I haven't checked everything, but it sure looks like we can go on indefinitely. $$ N=\langle a, a^{1/2},a^{1/4},a^{1/8},\ldots\rangle\unlhd G, $$ and then if you can show that $G=N\rtimes \langle b\rangle$ you are done, I think.