Let $G=\begin{bmatrix}a&b\\0&d\end{bmatrix}$ exist in $GL(2,\mathbb R)$.
So in order to prove this is a subgroup we need to show $G$ is closed with respect to $*$ and we also need to show $G$ is closed with respect to taking inverses.
I know $$\begin{bmatrix}a&b\\0&d\end{bmatrix}^{-1}=\begin{bmatrix}1/a&1/b\\0&1/d\end{bmatrix}$$ which exists in $GL(2,\mathbb R)$ since these are all real numbers.
Now to show it is closed under $*$ we will do $$\begin{bmatrix}a&b\\0&d\end{bmatrix}*\begin{bmatrix}a&b\\0&d\end{bmatrix}=\begin{bmatrix}a^2&ab+bd\\0&d^2\end{bmatrix}.$$ Since this exists in $GL(2,\mathbb R)$ then it is closed under $*$. [Thus $G$ is a subgroup].
I'm not sure if this is right but I just used the defintion of a subgroup.
You are wrong about the inverse. It turns out that$$\begin{bmatrix}a&b\\0&d\end{bmatrix}^{-1}=\begin{bmatrix}\frac1a&-\frac b{ad}\\0&\frac1d\end{bmatrix}.$$But it is still an element of $G$ and therefore there is no problem here.
And in order to prove that $G$ is closed with respect to the product of matrices, you should take two elements of $G$:$$\begin{bmatrix}a&b\\0&d\end{bmatrix}.\begin{bmatrix}a'&b'\\0&d'\end{bmatrix}=\begin{bmatrix}aa'&ab'+bd'\\0&dd'\end{bmatrix}\in G.$$