If $a$, $b$, $x$, $y$ are rational numbers such that
$$(ay-bx)^2+4(a-x)(b-y) = 0 $$
then either (i) $x = a, y = b$ or (ii) $1-ab$ and $1-xy$ are squares of rational numbers. (Math. Trip. 1903)
(From MISCELLANEOUS EXAMPLES ON CHAPTER I, 21)
Do I need to learn some number theory first?
BTW, I did find out that $(ay-bx)^2+4(a-x)(b-y) = (ay+bx-2)^2-4(1-ab)(1-xy)$, just don't know how to continue.
On
I think a solution is given in Hardy's book. Anyway, I found this at http://quod.lib.umich.edu/u/umhistmath/acm1516.0001.001/40?q1=continuous
If we write $a-x=\xi$, $b-y=\eta$, we obtain $$a^2\eta^2+b^2\xi^2+(4-2ab)\xi\eta=0$$ Solving this equation for the ratio $\xi/\eta$ we find that $\xi/\eta$ (which we know to be rational) involves the quantity $$\sqrt{(2-ab)^2-a^2b^2}=2\sqrt{1-ab}$$ Hence $1-ab$ must be the square of a rational quantity. The only alternative is $\xi=\eta=0$.
But the equation given may also be written in the form $$x^2\eta^2+y^2\xi^2+(4-2xy)\eta\xi=0$$ Hence we deduce the same conclusion for $\sqrt{1-xy}$.
You have $$(ay-bx)^2 +4(a-x)(b-y)=0$$
On expanding, you get $$a^2 y^2 + y(4x-4a-2abx)+(b^2x^2+4ab-4bx)=0$$
This is a quadratic in $y$. The disriminant will have to be a square of a rational number. The discriminant simplifies to $$16(1-ab)(x-a)^2$$
The working:
$$(4x-4a-2abx)^2 -4a^2(b^2x^2+4ab-4bx)$$ $$=16x^2 +16a^2 +4a^2b^2x^2-32ax+16a^2bx-16abx^2-4a^2b^2x^2-16a^3b+16a^2bx$$ Taking $16$ common, we get $$16[(x-a)^2-ab(x-a)^2]$$ $$=16(x-a)^2(1-ab)$$
Clearly, either $x=a$, or $(1-ab)$ is the square of a rational number.
Similarly, the original expression can be made into a quadratic in $x$ to get $$16(1-xy)(y-b)^2$$ as the discriminant.