how to find positive integer $m,n $
$(m^2 + n^2 )(n^2-m)= 2mn(m+n^2) $
answer is $(36,18), (36, 12) $
$n$-th try........
$ gcd (m,n)= p$, then $m=ap, n= bp$
$(a^2 p^2 + b^2 p^2 )(b^2p^2 -ap)= abcp^2(ap+ b^2p^2)$
$(a^2+ b^2 )(b^2p-a) = 2ab (a+ b^2p ) $
$ gcd(a^2+ b^2,b)=1$, $ gcd(b^2p-a,b)=1$
so, $b=1$
$(a^2+1)(p-a)= 2a (a+p)$
$ p= a (\frac{a+1}{a-1} )^2 $
$p$ is integer, so, $a=2,3$,
$p=n=18,12$
is this possible? and is there some relationship with pythagorean triple?
$(36, 18)$ and $(36, 12)$ are the only positive integer solutions.
Expand the equation and move all the terms to one side: $$m^3+mn^2+2m^2n-m^2n^2-n^4+2mn^3=0\tag{1}$$
Let $p=gcd(m,n)$ and $m=ap, n=bp$. Then by rewriting as a cubic equation in $a$, we get
$$a^3-(b^2p-2b)a^2+(2b^3p+b^2)a-b^4p=0$$
By the Rational Root Theorem, if $a$ is an integer, $a|b^4p$. Since $gcd(a,b)=1$, then $a|p$.
Now go back to equation $(1)$, and notice that it has some nice patterns that look like the $(x+y)^2=x^2+2xy+y^2$ formula. It can be rewritten as $$m(m^2+n^2+2mn)-n^2(m^2+n^2-2mn)=0$$ $$m(m+n)^2-n^2(m-n)^2=0$$ $$m(m+n)^2=n^2(m-n)^2$$ Now sub in $ap$ and $bp$ again: (The reason I didn't sub them in earlier is because it makes the equation more cluttered and makes it a lot harder to see patterns in the equation, like this square-of-sum formula.) $$a(a+b)^2=pb^2(a-b)^2\tag{2}$$ We know $a|p$, so $$\frac{p}{a}=\frac{(a+b)^2}{b^2(a-b)^2}=\left(\frac{a+b}{b(a-b)}\right)^2\in\mathbb{N}$$ so $$b(a-b)|(a+b)$$ Since $gcd(b,(a+b))=1$, then $b$ must equal 1. That leaves us with $(a-1)|(a+1)$, so $a$ can only be 2 or 3. Substituting these two possibilities back into $(2)$ quickly yields $p=18$ or $p=12$ respectively, so $(m,n)=(36,18)$ or $(36,12)$.
As for the Pythagorean triples, I'm not sure if there is any connection, since the formula is $$a=m^2-n^2$$ $$b=2mn$$ $$c=m^2+n^2$$ so $b$ and $c$ appear, but $a$ does not.