Let $G$ be the subgroup of $GL_2(\mathbb{R)}$ generated by $ \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$ and $ \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix}$.
Let $H$ is generated by $ \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$
I want to prove that $H$ is not quasi-isometric embedding to $G$.
My attempt: Let $ a=\begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$ and $b = \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix}$. Now I have found a relation $bab^{-1} = a^2$.
Now I cannot proceed further. Please help.
Ok, here it goes. The formula $$ b a b^{-1}=a^2, $$ and its consequence $b a^m b^{-1}=a^{2m}$, applied inductively yield: $$ b^n a b^{-n}=a^{(2^n)}, n\ge 0. $$ (This is where you have miscalculated. I will leave it to you to check the above formula using induction.)
For $w\in H$ let $|w|_H$ denote the distance of $w$ in $H$ to the neutral element in the group $H=\langle a\rangle$ with respect to the generating set $\{a^{\pm 1}\}$. Similarly, define $|u|_G$ for $u\in G$ with respect to $\{a^{\pm 1}, b^{\pm 1}\}$.
Then we get: $$ |a^{2^n}|_H= 2^n, |a^{2^n}|_G\le 2n+1. $$ Since $$ \lim_{n\to\infty} \frac{2n+1}{2^n}=0, $$ we conclude that there are no constants $K\ge 1, A\ge 0$ such that $$ K^{-1} |w|_H - A\le |w|_G $$ for all elements $w\in H$. Hence, the inclusion map $H\to G$ is not a quasi-isometric embedding.