Find all pairs of nonzero integers $(a,b)$ such that $(a^2+b)(a+b^2)=(a-b)^3$
My effort
Rearranging the equation I have \begin{array} \space (a^2+b)(a+b^2)-(a-b)^3 &=0 \\ a^2(b^2+3b)+a(-3b^2+b)+2b^3 &=0 \\ \end{array}
Letting $a=x$, we have the polynomial $Q(x)$ such that $$Q(x)=x^2(b^2+3b)+x(-3b^2+b)+2b^3 $$
So I must have that the roots $a_1,a_2$ must be such that \begin{array} \space a_1+a_2=-\cfrac{-3b+1}{b+3} \\ a_1\cdot a_2=\cfrac{2b^2}{b+3} \\ \end{array}
I am stuck now,should I go by brute force and verify some values or there's still something I can do here to simplify the problem ?
The general solution to $\alpha x^2+\beta x+\gamma=0$ is $x=\frac{-\beta\pm \sqrt{\beta^2-4\alpha\gamma}}{2\alpha}$.
Here, after factoring out the common $b$, you get $\alpha = b+3, \beta=-(3b-1),\gamma=2b^2$. In particular, you need $(3b-1)^2-4(b+3)(2b^2)=-8b^3-15b^2-6b+1$ to be a perfect square to even get a rational $a$.
But then a miracle occurs, because $-8b^3-15b^2-6b+1=(b+1)^2(1-8b)$. So this is a square exactly when $1-8b$ is a square.
Let $1-8b = (2c+1)^2$. Then $b=-\frac{c(c+1)}{2}$, and the formula:
$$ \begin{align}a&=\frac{3b-1 \pm (b+1)\sqrt{1-8b}}{2(b+3)}\\ &=\frac{3 \pm \sqrt{1-8b}}{2} +\frac{-10\pm 2\sqrt{1-8b}}{2(b+3)}\\ &=\frac{3\pm (2c+1)}{2} + \frac{-10 \pm 2(2c+1)}{-c^2-c+6}\\ &=\frac{3\pm (2c+1)}{2}+\frac{-10\pm(4c+2)}{(2-c)(c+3)} \end{align}$$
Note that $3\pm(2c+1)$ is even so you need the second have to be an integer.
But this just means you need $4c+12=4(c+3)$ divisible by $(c+3)(c-2)$, or you need $4c-8=4(c-2)$ divisible by $(c+3)(c-2)$. So you need either $c-2$ or $c+3$ a factor of $4$. We can restrict to $c$ non-negative, so we get $c=0,1,3,4,6$.
The case $c=0$ gives $b=0$, disallowed.
The case $c=1$ gives $a=b=-1$.
The case $c=3$ gives $a=9,b=-6$.
The case $c=4$ gives $(a,b)=(8,-10)$.
The case $c=6$ gives $(a,b)=(9,-21)$.
You should also check specifically the case where $b+3=0$, since then the equation is not quadratic. There is no solution in this case.
--
Wondering if we write the polynomial in terms of $b$:
$$2b^2 + (a^2-3a)b + (3a^2+a)=0$$
The discriminant of this is $a^4-6a^3-15a^2-8a$. While we have an easy factor of $a$ there, that just returns us to a cubic $a^3-6a^2-15a-8=(a+1)^2(a-8)$. But it isn't obvious to me how to see that factorization if you stumbled in this direction first.