Find all positive integers $n$ and $m$ such that $$(2mn+1)^2-4mn(m+n)+n^2+m^2+(n-1)^2+(m-1)^2-3$$ is a perfect square.
Of course $n=m=1$ is a trivial solution and if you put $n=m+1$ (assume WLOG that $n \geq m$), the above expression is equal to $4n^4$ which is a perfect square - But I don't know if this is the only solution. I would really appreciate any help.
On
(Too long for a comment) I think the question can be made clearer with a rearrangement. The $2(2mn)(m+n)$ smells like a quadratic: $$((m+n)-(2mn+1))^2=(2mn+1)^2-2(2mn+1)(m+n)+m^2+n^2+2mn$$ $$=(2mn+1)^2-4mn(m+n)-2(m+n)+m^2+n^2+2mn$$ Rearranging, $$=(2mn+1)^2-4mn(m+n)+m^2+n^2+2mn-2(m+n)$$ $$=[(2mn+1)^2-4mn(m+n)+m^2+n^2]+2mn-2(m+n)$$ And substituting, $$(2mn+1)^2-4mn(m+n)+m^2+n^2+(n-1)^2+(m-1)^2-3$$ $$=((m+n)-(2mn+1))^2-2mn+2(m+n)+(n-1)^2+(m-1)^2-3$$ $$=((m+n)-(2mn+1))^2-2mn+2m+2n+n^2-2n+m^2-2m-1$$ $$=((m+n)-(2mn+1))^2-2mn+n^2+m^2-1$$ $$=(m+n-2mn-1)^2+(n-m)^2-1$$
It's clear now why $n=m+1$ worked. It's equally clear that $m+n-2mn-1=1 \Rightarrow n=\frac{m-2}{2m-1}$ is a solution (although no positive integer solutions exist).
Given the OP's form,
$$(2mn+1)^2-4mn(m+n)+n^2+m^2+(n-1)^2+(m-1)^2-3 = w^2\tag{1}$$
ccorn and Alyosha showed it is equivalent to the more aesthetic forms,
$$\tfrac{1}{4}\big((2m-1)^2+1\big)\big((2n-1)^2+1\big)-1 =w^2$$
$$(m+n-2mn-1)^2+(n-m)^2-1 =w^2$$
respectively. There are infinitely many positive integer solutions to (1) other than $n=m+1$ as we can use a Pell equation $x^2-dy^2=1$ to solve it. In fact, we can use infinitely many $d$. For any positive integer $m>1$, the solution is then,
$$\begin{aligned} &n =(2m-3)x^2+4(m-1)^2xy-(m-2)\\ &w = 4(m-1)^2x^2+(2m-3)\big((2m-1)^2+1\big)xy-2(m-1)^2 \end{aligned}$$
where $x,y$ satisfy the Pell equation,
$$x^2-\big((2m-1)^2+1\big)y^2=1$$
For the simplest case when $m=1$, we have the alternative and simpler form,
$n = (x+1)/2,\;\; w=y,\;\; \text{where}\;\;x^2-2y^2=1.\tag{2}$
(Note: Since the $x$ of (2) is odd, then $n$ is an integer.)