Let $\mathbb{R}$ denote the set of real numbers. Find all functions $f:\mathbb{R}\rightarrow\mathbb{R}$ such that $$f(x^2)+f(xy)=f(x)f(y)+yf(x)+xf(x+y)$$ for all $x,y\in\mathbb{R}$.
If we put $x=0$ and mark $a=f(0)$ we get $$2a=af(y)+ya$$ so for $a\ne 0$ we get $f(y)=2-y$ for all $y$. Now say $a=0$, so $f(0)=0$. Leting $y=0$ we get $\boxed{f(x^2) = xf(x)}$. If we put this in starting equation we get $$xf(x) +f(xy)=f(x)f(y)+yf(x)+xf(x+y)\;\;\;\;(*)$$
From boxed equation we see that $f$ is odd: Edit $$-xf(-x) = f((-x)^2)= f(x^2)=xf(x) \implies f(-x)=-f(x)$$
Leting $y=-x$ in $(*)$ we get: $$xf(x)-f(x^2) =-f(x)^2-xf(x)\implies \underline{xf(x)= -f(x)^2}$$
Edit 28. 06. 2022
Let $G$ be a set of all real $x$ such that $f(x)=-x$. Clearly $0\in G$.
Suppose exists $x,y\in G$ so that $x \neq 0$. Here we can assume both $x,y$ are nonnegative, since $f(-x)=-f(x) = -(-x)$. Then we have $$-x^2+f(xy) =xf(x+y)$$ Then we have 4 possibilities:
- $f(xy)=0$ and $f(x+y)=0$ so $-x^2=0$ which is not true.
- $f(xy)=0$ and $f(x+y)=-x-y$ so $-x^2=-x^2-xy$ which is true only if $y=0$.
- $f(xy)=-xy$ and $f(x+y)=0$ so $-x(x+y)=0$ which is not true since $x,y$ are nonegative and $x\ne0$.
- $f(xy)=-xy$ and $f(x+y)=-x-y$ so $-x^2-xy=x(-x-y)$ which is true.
So if $x,y\in G$ then also $x+y\in G$ wich means $G$ is aditive subgroup of $(\mathbb{R},+)$ if$|G|\geq 2$.
What to do now? Is this $G$ usefull at all?
Let $P(x,y)$ be the assertion that $$f(x^2)+f(xy)=f(x)\,f(y)+y\,f(x)+x\,f(x+y)\,.$$ If $f(0)=0$, then we already know that $$f(x^2)=x\,f(x)=\big(f(x)\big)^2\text{ and }f(-x)=-f(x)$$ for all $x\in\mathbb{R}$.
The condition $P(x,x)$ implies that $$2\,x\,f(x)=f(x^2)+f(x^2)=\big(f(x)\big)^2+x\,f(x)+x\,f(2x)=2\,x\,f(x)+x\,f(2x)\,.$$ Thus, $x\,f(2x)=0$ for all $x\in\mathbb{R}$. That is, $$f(2x)=0\text{ for all }x\neq 0\,.$$ Since $f(0)=0$, we conclude that $$f(x)=0\text{ for all }x\in\mathbb{R}\,.$$
Thus, there are two possible solutions. First, as the OP has discovered, we have $f(x)=2-x$ for all $x\in\mathbb{R}$. Second, we have the trivial solution $f\equiv 0$.