Find all posssible $(a,b)$ where $a$ and $b$ natural number (positive integer) such that $a^2+a+3b$ and $b^2+b+3a$ both perfect square
so, i try to assume that $a^2+a+3b=(a+m)^2$ with $m$ natural number then we have $a+3b=2ma+m^2$ similarly $b+3a=2nb+n^2$
any further manipulation make the equation be more difficult non linear diophantine that i cannot solve no matter how many times i tried. please help
You made a good start. Here we set aside the trivial solution $a=b=0$. All you need to do next is check through a few possibilities for $m$ and $n$; most potential values for $m$ and $n$ can be ruled out, because the two equations $$3b=m^2+(2m-1)a\quad\text{and}\quad 3a=n^2+(2n-1)b$$ would then lead to contradictions. First note that, if $m\geqslant2$, then $2m-1\geqslant3$ and so $b>a$ by the first equation; similarly, if $n\geqslant2$, then $a>b$ by the second equation. Therefore, we cannot have both $m\geqslant2$ and $n\geqslant2$. Note also that $m$ (or $n$) cannot equal $2$, because $2^2$ is not divisible by $3$. So, by symmetry, we may assume $m=1$; the other solutions can be recovered by swapping the roles of $a$ and $b$. Our first equation becomes $a=3b-1$, and then $2(5-n)b= n^2+3$. By parity, $n\neq4$. Also, it is easy to check that $n\neq1$. We are left with $n=3$. Consequently the only solutions are $$\{a,b\}=\{3,8\}.$$