I am a beginner in Group Theory and I have found the following sencence in my Group Theory Lecture notes:
For another example, take $H$ the subgroup of $G=\langle a,b\mid a^2,b^2\rangle$ generated by $a$ and $(ab)^3$. Then the group $G \backslash H$ has exactly three elements $He, Hb$ and $H(ba)$ [...]
I am trying to understand what really means $G\backslash H$. It can easily show that $H$ is not normal in $G$, so the quotient $G/H$ cannot be constructed. Did $G\backslash H$ means the quotient of $G$ over the normalizer $N_G(H)$ of $H$?
I am not able to construct de normalizer of $H$ in order to check if $G/N_G(H)$ equals $\{He, Hb, H(ba)\}$.
Any help would be appreciated.
$\newcommand{\Span}[1]{\left\langle #1 \right\rangle}$It seems you are asked to find the distinct right cosets of $H$ in $G$.
First note that $G$ is the infinite dihedral group. Writing $c = a b$, an equivalent presentation is $$ G = \Span{c, a : a^{-1} c a = c^{-1}, a^2}, $$ showing that $G$ is a split extension of an infinite cyclic group $\Span{c}$ by a cyclic group $\Span{a}$ of order $2$, with $a$ acting on $\Span{c}$ by inversion.
So $\Span{c^{3}}$ is normal in $G$, and $G / \Span{c^{3}}$ is a dihedral group of order $6$. The rest should be clear.