$\text{SL}_2(\mathbb Z)$ acts on upper plane $\mathbb H= \{z \in \mathbb{C} | \Im(z) > 0 \}$ via Mobius transformation. $$ \text{ For } \gamma =\begin{bmatrix} a &b \\c&d \end{bmatrix} \in\text{SL}_2(\mathbb Z), \ \gamma z =\begin{bmatrix} a &b \\c&d \end{bmatrix}\cdot z = \frac{az +b}{cz+d} $$
Stabilizer of $z$ means set $\{\gamma \in \text{SL}_2(\Bbb Z), \gamma z=z\}$.
I want to know what kind of points have non-trivial stabilizer and the number of orbits.
My effort: For $z \in \Bbb H$, suppose $z = x + i y$.
$\text{For } \gamma =\begin{bmatrix} a &b \\c&d \end{bmatrix}, \gamma z =\frac{az+b}{cz+d}=z\iff az+b=cz^2+dz$
$$\iff ax+ayi+b = cx^2-cy^2+2cxyi + dx +dyi$$ $$\iff ay=2cxy+dy\ \&\ ax+b=cx^2-cy^2+dx$$ $$\iff a=2cx+d \ \&\ ax+b=cx^2-cy^2+dx$$ $$\implies b=-c(x^2+y^2),\ \gamma =\begin{bmatrix} 2cx+d &-c(x^2+y^2) \\c&d \end{bmatrix}.$$
$$\gamma \in \text{SL}_2(\Bbb Z),\ (2cx+d)\times d-(-c(x^2+y^2))\times c=1 $$ $$\implies (cx+d)^2+(cy)^2=1.$$
Then how to proceed? Thanks in advance.
I haven't learnt modular form yet, and I don't know if these help:
Good description of orbits of upper half plane under $SL_2 (Z)$
Orbit of complex unit $i$ under moebius tranformation in $SL_2(\mathbb{Z})$
Edit:
GTM$105$, Serge Lang, SL$_2(\mathbb R)$ might help.
Comment:
It's an exercise of section about group action on set, and before this section the book just introduces definition and basic property of group, so this problem is a bit more difficult than I thought.
I just realized that the points with stabilizers of order $2$ are in the same orbit, and the points with stabilizers of order $3$ are in the same orbit. There are therefore exactly two orbits with non-trivial stabilizers. The previous answer is too long and the browser is becoming slow, so I have to add another answer.
In the case of stabilizers of order $2$, recall that $c\mid a^2+1$. Using the knowledge about Gaussian integers (in particular, the unique factorization property), $c$ can be written as $$r^2+s^2=(r+si)(r-si)$$ for some $r,s\in\Bbb{Z}$ such that $r-si$ divides $a+i$. That is, $$\frac{a+i}{r-si}=p+qi$$ for some $p,q\in\Bbb{Z}$. Then observe that $$\begin{pmatrix}q&p\\s&r\end{pmatrix}\cdot i=\frac{p+qi}{r+si}=\frac{(p+qi)(r-si)}{r^2+s^2}=\frac{a+i}{r^2+s^2}=\frac{a}{c}+\frac{i}{c}.$$ Thus, $\frac{a}{c}+\frac{i}{c}$ is in the orbit of $i$.
In the case of stabilizers of order $3$, we can assume WLOG that we are in the case $d=-a+1$, where $c\mid a^2-a+1$. Let $\omega$ denote $-\frac12+\frac{\sqrt{3}i}{2}$. Using the knowledge about Eisenstein integers (in particular, the unique factorization property), $c$ can be written as $$r^2-rs+s^2=(r+s\omega)(r+s\overline{\omega})$$ for some $r,s\in \Bbb Z$ such that $r+s\overline{\omega}$ divides $a+\omega$. That is, $$\frac{a+\omega}{r+s\overline{\omega}}=p+q\omega$$ for some $p,q\in \Bbb Z$. Finally, observe that $$\begin{pmatrix}q&p\\s&r\end{pmatrix}\cdot\omega=\frac{p+q\omega}{r+s\omega}=\frac{(p+q\omega)(r+s\overline{\omega})}{r^2-rs+s^2}=\frac{a+\omega}{c}=\frac{2a-1}{2c}+\frac{\sqrt{3}i}{2c}.$$ Thus, $\frac{2a-1}{2c}+\frac{\sqrt{3}i}{2c}$ is in the orbit of $\omega$.