I cannot find a complete answer to the following problem (this is the source):
Q. Find all positive integers $(a,b)$ for which $(a+b)^2+4ab$ and $a^2+b^2$ are both squares.
Just something: clearly if $(a,b)$ works then $(a/c,b/c)$ works as well, where $c$ is the greatest common divisor of $(a,b)$. Hence we can assume they are coprime.
List of known primitive solutions: As remarked by Michael below, this is equivalent to solve the equation $(x^2-y^2)^2+12xy(x^2-y^2)+4x^2y^2=z^2$ with positive integers $x,y$. Adding the constraint $\mathrm{gcd}(x,y)=1$, by computer calculations we can see that all solutions $(x,y)$ with $x,y \le 30000$ are only the following ones: $(3,2)$, $(5,1)$, $(7,85)$, $(39,46)$, $(2717,1380)$, $(4097, 1337)$. This leads to the primitive solutions $$ (a,b)=(5,12), $$ $$ (a,b)=(5477689,7498920). $$
Edit (05 Dec 2015): the ones above are the unique primitive solutions also with the (last new) constraint $x,y \le 80000$..
In general, to make two quadratic polynomials as squares like,
$$a^2+b^2 = c^2\tag1$$
$$(a+b)^2+4ab = d^2\tag2$$
will yield one quartic to be made a square. Given the complete solution to Pythagorean triples as $a,b = m(x^2-y^2),\,2mxy$, then,
$$m^2(x^4 + 12 x^3 y + 2 x^2 y^2 - 12 x y^3 + y^4) = d^2$$
We can suppress $m,y$ without loss of generality,
$$u^4 + 12 u^3 + 2 u^2 - 12 u+1=v^2\tag3$$
Since you've found rational points to $(3)$ like $u =x/y = 3/2,\; 5/1,\; 7/85,\dots$, then it is birationally equivalent to an elliptic curve, so there is an infinite more (like $u = 616264191/46041814$). This yields,
$$a,b =377661704472473885,\;56747842513764948$$
Thus your system $(1)$, $(2)$ has an infinite number of primitive integer solutions, all of which are rational points on $(3)$.
P.S. For an elementary discussion on how to find more points on $(3)$, see this post.