Find all functions satisfying $f\left((1-xy)f(x)\right)+x^2f(y)=f(x)$

Question

To find all functions $f:\mathbb{R} \to \mathbb{R}$ satisfying $$f((1-x y) f(x))+x^2 f(y)=f(x) \label1\tag1$$ When $x=y=1$ we have $f(0)=0$.

Case I: If $f(x)$ is a constant function, say $f(x)=\lambda$, we have $$\lambda+x^2\lambda=\lambda$$ which makes sense only when $\lambda=0$. Thus the only constant function satisfying the functional equation is $$f(x)=0$$ Case II: If $f(x)$ is a non constant function, we have for $y=0$ in \eqref{1} $$f(f(x))=f(x)$$ Any help from here?

Answer 1

We'll show that, unless $f$ is the zero function, $f$ is injective; the result that $f$ must be the identity then follows from $f(f(x))=f(x)$.

First, we'll prove that $f(x_0)=0$ implies $x_0=0$ (call this property ($\star$)). If $x_0\in\mathbb R$ is such that $f(x_0)=0$, then substituting $x=x_0$ into the given equation gives that $x_0^2f(y)=0$. Since $f$ is not the zero function, there exists some $y$ for which $f(y)\neq 0$, and this implies $x_0=0$.

Now, take any real $r$ and $s$ for which $f(r)=f(s)$; we'll show that $r=s$. Property $(\star)$ proves this when $r$ or $s$ is zero, so we may assume $r$ and $s$ are nonzero.

Take any $x\neq 0$. By substituting $y=1/x$ in the given equation and using $f(0)=0$, we get $$x^2f(1/x)=f(x).\tag{1}$$ Now, let $a\neq 0$ be arbitrary, and substitute $y=\frac1{ax}$. We have $$f\big((1-1/a)f(x)\big)=f(x)-x^2f\left(\frac 1{ax}\right)=f(x)-a^{-2}(ax)^2f\left(\frac1{ax}\right)=f(x)-a^{-2}f(ax),$$ which rearranges to $$f(ax)=a^2f(x)-a^2f\big((1-1/a)f(x)\big).$$ In particular, substituting $x=r$ and $x=s$, we have $f(ar)=f(as)$ for every $a\neq 0$. Now, substitute $(1,ar)$ and $(1,as)$ into the given equation for some $a\neq 0$, and subtract the results. The $f(x)$ and $x^2f(y)$ terms cancel out, and we are left with $$f\big((1-ar)f(1)\big)=f\big((1-as)f(1)\big).$$ Taking $a=1/s$ gives that $$f\left(\left(1-\frac rs\right)f(1)\right)=f(0),$$ which implies that $(1-r/s)f(1)=0$. We know $f(1)$ cannot be $0$, so $r/s=1$, and $r=s$, as desired.

Answer 2

Let $P(x,y)$ be the claim $$f\left((1-xy)f(x)\right)+x^2f(y)=f(x).$$

As you already found $f(0)=0$ (for example from $P(1,1)$).

Now let $a\neq 0$ be such that $f(a)=0$, then $P(a,x)$ gives $a^2f(x)=0$ for all $x$, i.e. $f(x)=0$ for all $x$. So if any such $a\neq 0$ exists, then the function is necessary the zero function, which gives one solution. Otherwise $f(x)=0$ has the unique solution ($x=0$).

So assume $x=0$ is the unique zero of $f$. Take any $x\neq 0$, then $f(x)x\neq0$ and we are justified to set $P(x,\frac{f(x)-x}{f(x)x})$ to get $$ f(x)+x^2f(\frac{f(x)-x}{f(x)x})=f(x). $$ Then $$ f(\frac{f(x)-x}{f(x)x})=0. $$ By the assumption we must have $\frac{f(x)-x}{f(x)x}=0$ and hence $f(x)=x$. Since $x\neq 0$ was arbitrary and $f(0)=0$, we found a second solution $f(x)=x$ for all $x$ (the identity).

Plugging both $f(x)=0$ and $f(x)=x$ to the original equation we verify they are indeed solutions and by the above the only ones.

Note: The key substitution $y=\frac{f(x)-x}{f(x)x}$ is found simply by solving $(1-xy)f(x)=x$ for $y$, so that we get $f(x)$ that cancels on the other side of the equation.