Let $G:= \bigg\{\left( \begin{array}{ccc} a & b \\ 0 & a \\ \end{array} \right)\Bigg| a,b \in \mathbb{R},a\ne 0\bigg\}$. Let $H:= \bigg\{\left( \begin{array}{ccc} 1 & b \\ 0 & 1 \\ \end{array} \right) \Bigg| b \in \mathbb{R}\bigg\}$.
I know that $H$ is normal subgroup of $G$. I need to prove that $G/H\cong\mathbb{R}^*$.
My attempt:
$$G/H=\{gH\mid g\in G\}$$
$$ \left( \begin{array}{ccc} a & b \\ 0 & a \\ \end{array} \right) $$ $$ \left( \begin{array}{ccc} 1 & b \\ 0 & 1 \end{array} \right) = \left( \begin{array}{ccc} a & ab+b \\ 0 & a \end{array} \right) $$
And there are $\infty$ solutions from the form :$ \left( \begin{array}{ccc} a & ab \\ 0 & a \end{array} \right) $.
$$ \begin{vmatrix} \left( \begin{array}{ccc} a & ab \\ 0 & a \end{array} \right)\end{vmatrix}=\mathfrak{c}=|\mathbb{R}^*| $$
$$\Rightarrow G/N\cong \mathbb{R}^*$$
Is it correct? is there another way to solve this?
I don't quite understand your solution, but here is another way to do it:
You can get the isomorphism from the First Isomorphism Theorem. For example, consider the map $$ \phi: G \to \mathbb{R}^\times $$ given by $$ \phi\pmatrix{a & b \\ 0 & a} = a. $$ You have to show that this map
Then the First Isomorhpism Theorem will tell you that $G / H \simeq \mathbb{R}^\times$.
If you don't know the First Isomorphism Theorem, you can define the map $$ \psi: G/H \to \mathbb{R}^\times $$ by $$ \psi \pmatrix{a & b \\ 0 & a}H = a $$ and show that this map is
(Is you do this, then you pretty much prove the First Isomorphism Theorem in this special case.)