Which linear transformations preserve this?

Question

Let $a,b,x,y\in \mathbb{Z}$ (with $a,b$ given) and consider the equation $a(x^2+y^2)=bxy$. Consider transformations taking $x$ to $px+p'y$ and $y$ to $qx+q'y$. For which integers $p,q,p',q'$ is it true that the resulting equation (after expanding the substitutions) is of the form $A(x^2+y^2)=Bxy$?

Answer 1

It is clear that the transformed equation will always be of the form $Ax^2+Bxy+Cy^2=0$, so the real question is whether $A=C$.

Generally, an equation of the form $ax^2+bxy+cy^2=0$ has as its solution two lines through the origin (in addition to a few degenerate cases that are not really interesting here), and $a=c$ if and only if the solution is symmetric under a reflection through across line $x=y$.

The linear transformation $$ \begin{pmatrix}x\\y\end{pmatrix} \mapsto \begin{pmatrix} p & p' \\ q & q' \end{pmatrix} \begin{pmatrix}x\\y\end{pmatrix} $$ preserves this symmetry of two lines iff either it maps each of the lines $x=y$ and $x=-y$ to itself, or it maps one to the other and vice versa.

So a $(p,q,p',q')$ that preserve your form for all $a$ and $b$ must satisfy either $$ p+p' = q+q' \text{ and } p-p'=-(q-q') \qquad\text{so}\qquad p=q' \text{ and } p'=q $$ or $$ p+p' = -(q+q') \text{ and } p-p'=q-q' \qquad\text{so}\qquad p=-q' \text{ and } p'=-q $$