Find all function that satisfy $(f(x) + xy) \cdot f(x - 3y) + (f(y) + xy) \cdot f(3x - y) = (f(x + y))^2$

Question

Find all functions $f: \mathbb{R} \to \mathbb{R}$ such that for all real numbers $x$ and $y$,$$(f(x) + xy) \cdot f(x - 3y) + (f(y) + xy) \cdot f(3x - y) = (f(x + y))^2.$$

Answer 1

Letting $x=y=0$, we have $ f(0)^2+f(0)^2=f(0)^2$, hence $$\tag1f(0)=0.$$ If $a=x-3y$ and $b=3x-y$ then $x+y=\frac{b-a}2$, whence $$\tag2f(a)=f(b)=0\implies f\left(\frac{b-a}2\right)=0$$ and in combination with $(1)$ $$ \tag3f(x)=0\implies f\left(\pm\tfrac12x\right)=0.$$ Assume $f(a)=0$ (with $a\ne 0$). Then for $x=\frac12a$, $y=-\frac12a$, we have $f(x)=f(y)=0$ by $(3)$ and so $2xyf(2a)=f(0)^2$, i.e., $$\tag4f(a)=0\implies f(2a)=0. $$ From $(1)$, $(2)$, and $(4)$ we see that $A:=\{\,x\in\Bbb R\mid f(x)=0\,\}$ is a subgroup of $\Bbb R$, which is closed under division by $2$ because of $(3)$.

Let $B=(\Bbb R\setminus A)\cup \{0\}$. As $A$ is a group and $(3)$, we have that $$\tag{$\star$} a\in A, b\in B\implies -b\in B,2b\in B,\frac12b\in B,b\pm a\in B.$$

For $y=-x$, we have $$(f(x)-x^2)f(4x)+(f(-x)-x^2)f(4x)=f(0)^2,$$ so (using $f(4x)=0\iff f(\pm x)=0$) $$ \tag5x\in B\implies f(x)+f(-x)=2x^2.$$ Letting $x=y$, we have $$(f(x)+x^2)(f(-2x)+f(2x))=f(2x)^2,$$and likewise ($x\leftrightarrow-x$) $$(f(-x)+x^2)(f(-2x)+f(2x))=f(-2x)^2,$$ hence by subtracting $$(f(x)-f(-x))(f(-2x)+f(2x))=f(2x)^2-f(-2x)^2. $$ If $x\notin A$, we can divide off $f(-2x)+f(2x)=8x^2$ to obtain $$\tag6 f(x)-f(-x)=f(2x)-f(-2x)$$ (which trivially also holds for $x\in A$). Adding $8x^2$ and using $(5)$ $$\tag7 x\in B\implies f(x)+3x^2=f(2x)$$ Revisiting the case $y=x\in B$ now gives $$\begin{align}x\in B\implies\quad (f(x)+x^2)(f(2x)+f(-2x))&=f(2x)^2\\ (f(x)+x^2)8x^2&=(f(x)+3x)^2\\ 8x^2f(x)+8x^4&= f(x)^2+6x^2f(x)+9x^4\\ (f(x)-x^2)^2&=0\end{align} $$ i.e., $$x\in B\implies f(x)=x^2.$$

Assume $x,y\in B$ and $x+y\in A$. Then by $(\star)$, $$3x-y=2\cdot2\cdot x-(x+y)\in B$$ and likewise $x-3y\in B$, so that $$f(x+y)^2=(x^2+xy)(3x-y)^2+(y^2+xy)(x-3y)^2=(x+y)^4,$$ i.e., also $x+y\in B$. We conclude that $B$ is also a group. But the union of two subgroups is the whole group $\Bbb R$ only if one already equals $\Bbb R$. Therefore $A=\Bbb R$ or $B=\Bbb R$, i.e., the only two solutions $f_1,f_2$ of the functional equation are $f_1(x)=0$ and $f_2(x)=x^2$.