I would like to know the subgroup generated by the following type of matrices.
$$A=\left(\begin{matrix}1&a\\ 0&1\end{matrix}\right)$$ and $$B=\left(\begin{matrix}1&0\\ b&1\end{matrix}\right)$$
for integer $a,b\in\mathbb Z$.
Is there a way we can parameterize the full set of matrices $G\in\mathcal G=\langle A,B\rangle$?
The group operation is the matrix product
On
From the following reference
https://kconrad.math.uconn.edu/blurbs/grouptheory/SL(2,Z).pdf
we see the group $\text{SL}(2,\mathbb Z)$ is generated by the matrices
$$S=\left(\begin{matrix}0&-1\\1&0\end{matrix}\right)\quad\text{and}\quad T=\left(\begin{matrix}1&1\\0&1\end{matrix}\right)$$
We can see that the matrices given in the question are enough to generate $T$ and $S$ and hence they also generate the full group $\text{SL}(2,\mathbb Z)$.
$T$ can be identified as $T=A(a=1)$.
$S$ can be created using the matrices $A(a=-1)$ and $B(b=1)$, i.e.
$$S=A(a=-1)B(b=1)=\left(\begin{matrix}1&-1\\0&1\end{matrix}\right)\left(\begin{matrix}1&0\\1&1\end{matrix}\right)=\left(\begin{matrix}0&-1\\1&0\end{matrix}\right)$$
$\newcommand{\Z}{\mathbf{Z}}$Here's a full proof that the subgroup $\Gamma$ generated by the matrix $u=\begin{pmatrix}1 & 1\\0 & 1\end{pmatrix}$ and its transpose $v$ is all of $\operatorname{SL}_2(\Z)$. Given a matrix $m=\begin{pmatrix}a&b\\c&d\end{pmatrix}$, we have $$mu=\begin{pmatrix}a&b+a\\c&d+c\end{pmatrix},mu^{-1}=\begin{pmatrix}a&b-a\\c&d-c\end{pmatrix},mv=\begin{pmatrix}a+b&b\\c+d&d\end{pmatrix},mv^{-1}=\begin{pmatrix}a-b&b\\c-d&d\end{pmatrix}.$$ Write $N(m)=|a|+|b|$. We see that as soon as $|ab|>0$, one of these four matrices $m'$ satisfies $N(m')<N(m)$.
Hence, choosing $m$ that minimizes $N$ within $m\in m\Gamma$, we see that $N(m)=0$. Since entries in the first row are coprime (because determinant is $\pm 1$), the first row of $m$ is $(\pm 1,0)$ or $(0,\pm 1)$. Using the elementary operations above (which remain in $m\Gamma$), we can change $$(-1,0)\to (-1,1)\to (0,1),\quad (0,1)\to (1,1)\to (1,0),\quad (0,-1)\to (1,-1)\to (1,0).$$ Hence $m\Gamma$ contains a matrix with first row $(1,0)$, so, the determinant is $1$, this matrix has the form $\begin{pmatrix}1&0\\c&1\end{pmatrix}$, namely equals $v^c$. Hence $v^c\in m\Gamma$, and since $v\in\Gamma$ this ensures $m\in\Gamma$.