We consider the natural action of $\operatorname{PSL}_2(\mathbb{Z})$ on a positive definite binary quadratic form $Q$ given by$$\begin{pmatrix}p & q\\ r & s \end{pmatrix}Q(X,Y)=Q(pX+qY,rX+sY). $$
Don Zagier mentions the following fact about the stabilizer $\operatorname{PSL}_2(\mathbb{Z})_Q$ in his paper "Traces of singular moduli" (p. 2 after equation (2)):
$$ |\operatorname{PSL}_2(\mathbb{Z})_Q|=\begin{cases} 3 & \text{if $Q$ is $\operatorname{PSL}_2(\mathbb{Z})$-equivalent to }\tilde{Q}(X,Y)=aX^2+aXY+aY^2\\ 2 & \text{if $Q$ is $\operatorname{PSL}_2(\mathbb{Z})$-equivalent to }\tilde{Q}(X,Y)=aX^2+aY^2\\ 1 & \text{else } \end{cases} $$
I'm trying to understand how Zagier calculated those numbers. My first idea is to compare coefficients and solve the equation system, but this seems rather tedious. Is there another way to derive the above-mentioned result?
I am very grateful for any help,
Sincerely, Hypertrooper
Each positive definite form can only have a finite number of matrices stabilising it, so each such must have finite order.
In $\text{SL}_2(\Bbb Z)$ your matrix $A=\pmatrix{p&q\\r&s}\ne\pm I$ can only have finite order iff its entries are roots of unity. So its trace is $\le2$ in absolute value. All $A$ with trace $0$ are conjugate to $\pmatrix{0&1\\-1&0}$ and those with trace $\pm1$ to $\pm\pmatrix{0&1\\-1&-1}$. In the first case $V$ stabilises only forms $a(X^2+Y^2)$ and in the second case they stabilise only forms $a(X^2+XY+Y^2)$. Replacing $V$ by a conjugate replaces the form by an equivalent form.