The functional equation $ f ^ 4 \left( x f ( x ) ^ 2 + y f ( x ) - x \right) = x ^ 3 + x f ^ 2 ( y ) - x $

Question

For any $ f : \mathbb R \to \mathbb R $ and any nonnegative integer $ n $, let $ f ^ n $ denote the $ n $-th iteration of $ f $; i.e. $ f ^ 0 ( x ) = x $ and $ f ^ { n + 1 } ( x ) = f \big( f ^ n ( x ) \big) $ for all $ x \in \mathbb R $.

Find all functions $ f : \mathbb R \to \mathbb R $ satisfying $$ f ^ 4 \left( x f ( x ) ^ 2 + y f ( x ) - x \right) = x ^ 3 + x f ^ 2 ( y ) - x \tag 0 \label 0 $$ for all $ x , y \in \mathbb R $.

It's straightforward to check that the identity function is a solution. I suspect that it's the only solution. One witness to this is that assuming $ f $ is real analytic, some messy calculations show that identity is the only candidate. I don't know whether there are non-analytic solutions or not. I just have the following observations.

Plugging $ x = y = 0 $ in \eqref{0} we get $ f ^ 4 ( 0 ) = 0 $. Then, putting $ x = 0 $ and $ y = 1 $ in \eqref{0} we get $ f ^ 5 ( 0 ) = 0 $, and since $ f ^ 5 ( 0 ) = f \left( f ^ 4 ( 0 ) \right) $, we have $ f ( 0 ) = 0 $. Hence, setting $ y = 0 $ in \eqref{0} we get $$ f ^ 4 \left( x f ( x ) ^ 2 - x \right) = x ^ 3 - x \tag 1 \label 1 $$ for all $ x \in \mathbb R $. In particular, \eqref{1} shows that $ f $ is surjective, since $ x ^ 3 - x $ takes every real number as value. As a special case, we must have $ f ( x _ \pm ) = \pm 1 $ for some $ x _ \pm $, and letting $ x = x _ \pm $ in \eqref{1} we get $ x _ \pm \in \{ - 1 , 0 , 1 \} $. As we know that $ f ( 0 ) = 0 \ne \pm 1 $, $ 0 $ is excluded, and because both $ - 1 $ and $ 1 $ must be covered, we get $ f ( - 1 ) = - f ( 1 ) \in \{ - 1 , 1 \} $.

Source:

Number four on the list at the end of this page.

Answer 1

You've got everything you need to finish. Here's some hidden hints, as to not spoil the solution.

We know that $f(1)\in \{-1,1\}$. First investigate what happens if $f(1)=1$.

In this case, try to prove that $f(f(x))=x$ for all real $x$. If we get something like $f^4(y)=f^2(y)$, this follows by surjectivity (why?). Then...?

From the above, see if you can deduce what $f$ must be. So all that's left...

We're left with the case where $f(1)=-1$. What happens if you do the same thing as in the $f(1)=1$ case? Do we find another solution, or can we derive a contradiction?

---

For completeness, I'll put a full solution below (continued from your progress).

Suppose that $f(1)=-1$. Then if we take $x=1$, we obtain $f^4(-y) = f^2(y)$. Then taking $y \to -y$, we obtain $f^4(y)=f^2(-y)$, whereupon iterating $f$ twice on both sides gives $$f^6(y) = f^4(-y)=f^2(y).$$ By surjectivity, $f(f(y))$ can assume any real value, and hence $f^4(y)=y$. Thus $f(f(-y))=y$, or $f(f(y))=-y$ for all real $y$. The original equation then becomes $$xf(x)^2+yf(x)-x = x^3-xy-x,$$ from which it's easy to see that $f(x)=-x$ for all real $x$. But this contradicts $f(f(y))=-y$, so $f(1)$ cannot be $-1$.

So instead assume that $f(1)=1$. Taking $x=1$ in the equation, we immediately obtain $f^4(y)=f^2(y)$, implying that $f(f(x))=x$ for all real $x$ by surjectivity. It's straightforward to deduce from the original equation that this leads to $f(x)=x$, which is indeed a solution.

Hence the identity function is the only satisfying one.