Given that $a$, $b$ are integers with $a>b$ and the two roots $\alpha$, $\beta$ of the equation $$3x^2+3(a+b)x+4ab=0$$ satisfy the relation $$\alpha(\alpha+1)+\beta(\beta+1)=(\alpha+1)(\beta+1)$$ find all the pairs $(a,b)$ of two integers.
I can't seem to approach it properly.
On
Because $\alpha$ and $\beta$ are roots of $3x^2+3(a+b)x+4ab$, Vieta says $$ \alpha+\beta=-a-b\quad\text{and}\quad\alpha\beta=\frac43ab\tag1 $$ Since $\alpha(\alpha+1)+\beta(\beta+1)=(\alpha+1)(\beta+1)$, we have $$ (\alpha+\beta)^2=3\alpha\beta+1\tag2 $$ Plugging $(1)$ into $(2)$, we have $$ (a+b)^2=4ab+1\iff(a-b)^2=1\tag3 $$ Since $a\gt b$, we get $$ a=b+1\tag4 $$
Start from $$\alpha(\alpha+1)+\beta(\beta+1)=(\alpha+1)(\beta+1)$$ Expand the parenthesis, and move all terms to one side: $$\alpha^2+\alpha+\beta^2+\beta-\alpha\beta-\alpha-\beta-1=0$$ I am going to write this in terms of the sum of the roots, $\alpha+\beta$, and the product. $$\begin{align}\alpha^2+\alpha\beta+\beta^2-1&=0\\\alpha^2+2\alpha\beta+\beta^2-\alpha\beta-1&=0\\(\alpha+\beta)^2-\alpha\beta-1=0\end{align}$$ From Vieta's formula, $$\alpha+\beta=-\frac{3(a+b)}3=-(a+b)$$ and $$\alpha\beta=\frac{4ab}3$$ From here $$[-(a+b)]^2-\frac{4ab}3-1=0$$ $$a^2+\frac {2b}3a+(b^2-1)=0$$ $$a_{1,2}=-\frac{b}3\pm\sqrt{\frac{b^2}9-(b^2-1)}=-\frac{b}3\pm\sqrt{1-\frac{8b^2}{9}}$$ For $a$ to be real, $b=0$ or $b=1$. If $b=0$ then $a=\pm1$. Since you are told $a>b$, it means only $a=1$ is good. If $b=1$, $a=-\frac 13\pm\frac13$. One solution is not integer, the other is less than $b$.
Note: As mentioned in the comments, there is a sign mistake in the sign after the second equation. The third equation should be $$\alpha^2+\beta^2-\alpha\beta-1=0$$ After that $$(\alpha+\beta)^2-3\alpha\beta-1=0$$ With the Vieta, $$[-(a+b)]^2-3\frac{4ab}3-1=0\\(a-b)^2=1$$ Since $a>b$, then $$a=b+1$$