An integral quadric form is some instance of $f(x,y)=ax^2+bxy+cy^2$, with $a,b,c$ integers. Let $f(x,y),g(x,y)$ be two integral quadric forms. Then we say that they are improperly equivalent, denoted by $f(x,y) \sim_{imp} g(x,y)$, if there exist integers $p,q,r,s$ such that
$f(x,y)=g(px+qy,rx+sy)$ and $ps-qr=-1$
I am trying to prove that improper equivalence is not an equivalence relation, i.e., it is not reflexive, symmetric, and transitive. To do so, I would need to find some counterexample to reflexivity, symmetry, or transitivity, which would look like:
Reflexivity fails: $f(x,y) \not \sim_{imp} f(x,y)$
Symmetry fails: $f(x,y) \sim_{imp} g(x,y)$, but $g(x,y) \not \sim_{imp} f(x,y)$
Transitivity fails: $f(x,y) \sim_{imp} g(x,y)$ and $g(x,y) \sim_{imp} h(x,y)$, but $f(x,y) \not \sim_{imp} h(x,y)$.
I am not sure how to find such counterexamples, or whether any of these properties fail in general. I have been able to prove that proper equivalence (defined the same as improper, except that $ps-qr=1$) is an equivalence relation, but I seem unable to figure out how to show that improper equivalences are not an equivalence relation. Any hints are appreciated.
It's easiest to find a counterexample to reflexivity. We only need to find a form that doesn't have an improper equivalence to itself (also called an improper automorphism).
One such example would be $f(x, y) = 2 x^2 + x y + 3 y^2$. Suppose an automorphism is given by $$f(px+qy,rx+sy) = (2p^2+p r + 3r^2)x^2 + (4pq +ps + qr + 6 rs) xy + (2 q^2+qs+3s^2) y^2.$$ Then $2p^2+p r + 3r^2 = 2$. Since $$2 p^2 + p r + 3 r^2 = 2 \left(p+\frac{r}{4}\right)^2 + \left(3-\frac{1}{8}\right) r^2 > 2 r^2, $$ we must have $r = 0$. This also implies that $p = \pm 1$. To make sure that this is an equivalence, we need $s = \pm 1$. Matching the coefficient of $y^2$ forces $q$ to be zero. However, the coefficient of $xy$ now reduces to $ps$, which must equal $1$, but this cannot be satisfied if we want the equivalence to be improper.
Of the other two conditions for equivalence relations, symmetry is actually always guaranteed, because if $f(x, y) = g(p x + q y, r x + s y)$ and $ps - qr = -1$, then $g(x, y) = f(-s x + q y, r x - p y)$.
Now that we have a counterexample to reflexivity, it is not hard to construct a counterexample to transitivity. You can use the $f$ above, take any improper equivalence transform to $g(x, y) = f(p x + q y, r x + s y)$ so that $f\sim_\text{imp} g$ and $g\sim_\text{imp} f$, but $f\not \sim_\text{imp}f$.
This counterexample is inspired by Theorem 2.4 in Binary Quadratic Forms by Duncan A. Buell. The basic idea is to work with reduced positive definite forms. Given $f(x, y) = a x^2 + b x y + c y^2$, it is definite if the descriminant $\Delta = b^2 - 4 a c$ is negative, and reduced if $|b| \leq a \leq c$. These conditions are strong enough to restrict the automorphism to only a few cases.