$\textbf{Question:} \\ $ Find all functions $f : N \to N$ for which the expression $af(a) + bf(b) + 2ab$ is a perfect square for all $a,b \in \mathbb N$
$\textbf{My progress:}$
1.$ p \mid f(np) $ for all primes p and positive integer n.
2.$f(1)$ is a quadratic residue modulo infinitely many primes which (probably) implies $f(1)$ itself is a perfect square
3.all primes other than 2 can occur only once in the prime factor decomposition of $f(1)$
4.2 and 3 together gives $f(1)=2^k$. for even k. Then using $2*f(1)+2$ being a perfect square we get $f(1)=1$
5.And I also noticed $f(a)=a$ works.
Here N means only the positive whole numbers.
It's not true that if $m$ is a quadratic residue modulo infinitely many primes, then $m$ must be a square. For example, $2$ is a quadratic residue modulo every prime that is congruent to $\pm 1$ modulo $8$, but $2$ is not a square.
Fortunately, it is true that if $m$ is a quadratic residue modulo every sufficiently large prime, then $m$ is a square.
For any natural number $a$, and any prime $p$, we have that $$ af(a) \equiv af(a) + pf(p) + 2ap \pmod p $$ and so $af(a)$ is a square modulo every prime $p$. It follows that $af(a)$ is a square for every natural number $a$.
We will show that if $p$ is an odd prime then $f(p) = p$.
Suppose that $q \neq p$ is an odd prime such that $q \mid f(p)$.
Then $ qf(q) + pf(p) + 2pq $ is a square that is divisible by $q$, and hence by $q^2$. We note that $qf(q)$ is divisible by $q^2$, and so if $f(p)$ were divisible by $q^2$, then $2pq$ would be divisible by $q^2$, which is a contradiction.
Thus $f(p)$, and hence $pf(p)$, is divisible by $q$ only once, which is a contradiction since $pf(p)$ is a square.
Thus the only primes that divide $f(p)$ are $2$ (possibly), and $p$. Since $pf(p)$ is a square, we know that $p$ divides $f(p)$ an odd number of times.
Note that $$ 2pf(p) + 2p^2 $$ is a square. If $f(p)$ were even, then the left hand side would be $2$ modulo $4$, which would be a contradiction. Thus, in fact, $2$ does not divide $f(p)$. We thus have that $f(p) = p^{2t + 1}$ for some natural number $t$.
We now note that $$ p^{2t + 2} + 2p + 1 = pf(p) + 1f(1) + 2p $$ is a square. But $$ (p^{t + 1})^2 < p^{2t + 2} + 2p + 1 \leq p^{2t + 2} + 2p^{t + 1} + 1 = (p^{t + 1} + 1)^2 $$ and so we must have that $t = 0$ and $f(p) = p$.
We now have that for every natural number $a$, and every odd prime $p$, that $$ af(a) + 2ap + p^2 $$ is a square, and so is equal to $(p + m_p)^2$ for some $m_p$ which depends on $p$.
We easily see that $0 < m_p \leq af(a)$ for all $p$. (We could probably find a better upper bound, but the point is that it is bounded by something that does not depend on $p$.)
Thus there are infinitely many primes $p$ with the same value of $m_p$. Let this value be $m$.
Then there are infinitely many primes $p$ such that $$ af(a) + 2ap + p^2 = p^2 + 2mp + m^2 $$ or $$ af(a) + 2ap = m^2 + 2mp $$
Thus $af(a) - m^2$ is divisible by infinitely many primes $p$, and so $af(a) = m^2$. The equation then becomes $m^2 + 2ap = m^2 + 2mp$, and so $a = m$. Thus $af(a) = a^2$, and so $f(a) = a$ for all $a$.