$x^2+(a^2+b^2)x+ab=0$ where $a,b,x\in \mathbb Z$

Question

Trivial solutions include $a=0, x=-b^2$ and $b=0,x=-a^2$. Then $a=b=1,x=-1$ or $a=b=-1, x=-1$. Is there any other? How could you prove there aren't? $(a^2+b^2)^2-4ab=(a^2+b^2)+2ab(ab-1)$ needs to be a square. I tried to look at it $\mod 4$ but it doesn't yield anything useful.

Answer 1

Suppose $a,b\ne 0$ and $(a^2+b^2)^2-4ab = c^2$ for some integer $c\ge 0$. We can then write $c = a^2+b^2 - k$ for some $0<k\le a^2+b^2$. Then $$ 4ab = (a^2+b^2) - c^2 = (a^2+b^2) - (a^2+b^2-k) = k(2a^2+2b^2-k) $$ Note that, for $0<k\le a^2+b^2$, the RHS is increasing in $k$. Hence, if $k\ge 2$, we have $$ 4ab = k(2a^2+2b^2-k)\ge 4a^2+4b^2-4 \implies 2a^2+2b^2 - 4 + 2(a-b)^2\le 0. $$ This is false if $a^2+b^2> 2$, which occurs if at least one of $|a|$ or $|b|$ is greater than 1. In that case, $k = 1$, thus implying $$ 4ab = 2a^2+2b^2 - 1\implies 2(a-b)^2 - 1 = 0 $$ which is also impossible. Hence, there are no nontrivial solutions, where nontrivial here means $|ab|>1$.

---

Edit: Starting from the equality $4ab = k(2a^2+2b^2-k)$, another way to limit the possibilities is to note the following: for $k\ge 1$, we have $$ 4ab = k(2a^2+2b^2-k)\ge 2a^2+2b^2 - 1\implies 2(a-b)^2 - 1 \le 0.$$ Since $a$ and $b$ are integers, this can happen only if $a=b$, in which case we have $$ 4a^2 = k(4a^2 - k)\implies k^2 - 4a^2k + 4a^2 = 0.$$ The discriminant of the above quadratic is $$(4a^2)^2 - 4(4a^2) = 4a^2(4a^2-4)$$ which is positive if $a^2>1$, i.e. there are no solutions if $a^2>1$.