$1988$ IMO Problem 6 states:
Let $a$ and $b$ be positive integers such that $ab + 1$ divides $a^2 + b^2$. Show that $$\frac{a^2 + b^2}{ab + 1}$$ is the square of an integer.
My question is: what if $a$ and $b$ don't need to be positive? What other values of $\frac{a^2 + b^2}{ab + 1}$ are possible? I found a way to get $-5$ by setting $a = -1$ and $b = 2$. Is there a simple classification of such reachable integers?
I prefer to graph everything. The thing that Vieta Jumping does is to guarantee a point (if there are any integer points) on a short arc of the hyperbola. Then inequalities based on the size of $n$ tell us whether there can be any such integer points.
Here $$ \frac{x^2 + y^2}{xy-1} = n $$ with $x,y > 0.$
If there are any integer points at all, there are jumped integer points that lie on the arc between the endpoints $$ \left( \; \sqrt{\frac{4n}{n^2-4}} \; , \; \; \sqrt{\frac{n^3}{n^2-4}} \; \; \right) $$ and $$ \left( \; \sqrt{\frac{n^3}{n^2-4}} \; , \; \; \sqrt{\frac{4n}{n^2-4}} \; \; \right) $$
One must consider $n=3,4$ separately. For $n= 5$ there are indeed integer points at $(1,2) , \; \; (2,1).$
For $n \geq 6$ we see the picture below (done with $n=16.$ The axis of symmetry $y=x$ meets the hyperbola at $(t,t)$ where $t = \sqrt{ \frac{n}{n-2}},$ between $1$ and $2$
For $x \geq \sqrt{ \frac{n}{n-2}},$ but $x \leq \sqrt{\frac{n^3}{n^2-4}} , $ we see that $y$ is decreasing. Furthermore, for $x = 2$ we find $$ y_2 = \frac{n+4}{n+\sqrt{n^2 - n - 4}} .$$ Thus, for $$ 2 \leq x \leq \sqrt{\frac{n^3}{n^2-4}}$$ we see $0 < y \leq \frac{n+4}{n+\sqrt{n^2 - n - 4}} .$ For $n \geq 6$ this is smaller than $1$
So that's it: we can jump an integer point to an integer point along a certain bounded arc of the hyperbola. Then, for $n \geq 6$ we see there are no integer points within that arc. So, no integer points at all.