Let $\tau$ be a CM point in the upper half plane $\mathcal{H}$ - that is, an element of $\mathcal{O}_K$ for an imaginary quadratic extension $K/\mathbb{Q}$ that lies in $\mathcal{H}$ after choosing an embedding $K\hookrightarrow\mathbb{C}$. Equivalently, $\tau$ is a representative for the isomorphism class of a CM elliptic curve $E/\overline{\mathbb{Q}}$.
The following might be a basic/stupid question exposing a lack of understanding on my part, but has come up while I've been working on something else.
Let $\gamma\in SL_2(\mathbb{Z})$ be a matrix (in fact, we only need to consider its image in $PSL_2(\mathbb{Z})$). For which such CM points $\tau$ is $\gamma(\tau)$ also a CM point in $\mathcal{H}$?
My suspicion is that if both $\tau$ and $\gamma(\tau)$ are CM points then $\tau$ must be one of the three "elliptic points" (i.e. points whose stabiliser in $PSL_2(\mathbb{Z})$ is nontrivial):
$$\tau = i, \quad \rho = \frac{-1+i\sqrt{3}}{2}, \quad -\overline{\rho} = \frac{1+i\sqrt{3}}{2}.$$
It is not hard to see this for $\gamma = S = \left(\matrix{0 &-1\\ 1 & 0}\right)$: assuming that $\tau$ and $S(\tau) = -1/\tau$ are both CM points, we get that
$$N_{K/\mathbb{Q}}(\tau),N_{K/\mathbb{Q}}(S(\tau))\in\mathbb{Z}.$$
But $N_{K/\mathbb{Q}}(S(\tau)) = N_{K/\mathbb{Q}}(\tau)^{-1}$, so both norms are units in $\mathbb{Z}$ and therefore $\tau$ and $S(\tau)$ are in $\mathcal{O}_K^\times$. But the only imaginary quadratic fields $K$ for which (under some embedding $K\hookrightarrow \mathbb{C}$) the unit group $\mathcal{O}_K^\times$ intersects the upper half plane $\mathcal{H}$ are $K = \mathbb{Q}(i)$ and $K = \mathbb{Q}(\rho)$. Therefore, choosing such an embedding we get that $\tau$ must be one of $i$, $\rho$ or $S(\rho) = -\overline{\rho}$.
Does this hold for all $\gamma$? My intuition is that it does, but I am not at all sure why. I have the following vague idea: the moduli space of (complex) elliptic curves is $\mathcal{M}_{1,1} = \mathcal{H}//SL_2(\mathbb{Z})$ (viewed as a complex orbifold). Roughly, the points $x$ of this moduli space can be viewed as orbits $x = [\tau]$ of the action of $SL_2(\mathbb{Z})$ on $\mathcal{H}$, except at the "orbifold points" corresponding to $\tau = i$ and $\rho$, where the stabilisers are nontrivial. Something strange seems to be going on at these points.