Let $f:\Bbb{R}\to \Bbb{R}$, and $P(x)$, $Q(x)$ be given real coefficient polynomials, where the degree is odd and for every $x, y\in\Bbb{R}$: $$f(P(x)+Q(f(y))+Q(y)P(f(x)))=Q(y)+P(f(x))+P(x)Q(f(y))$$ Find $f(x)$.
I tried many times, I found the final result is only $f(x)=x$. Is it correct? If yes, how to prove it?
Let $P = a_m x^m + \cdots$, $Q = b_n x^n + \cdots$ and $f = c_p x^p + \cdots$, with $a_m, b_n, c_p \ne 0$ and $m, n$ odd. We shall look only at the leading terms, i.e. the terms of maximum degree in both sides of the equality (both sides being understood as elements of $\Bbb R [x,y]$).
In the left-hand side, you have
$$c_p \Big( (a_m x^m + \cdots) + \color{blue} {\big( b_n (f(y))^n + \cdots \big)} + (b_n y^n + \cdots) \color{green} {\big( a_m (f(x))^m + \cdots \big)} \Big) ^p + \cdots = \\ c_p \Big( (a_m x^m + \cdots) + \color{blue} {\big( b_n (c_p y^p + \cdots)^n + \cdots \big)} + (b_n y^n + \cdots) \color{green} {\big( a_m (c_p x^p + \cdots)^m + \cdots \big)} \Big) ^p + \cdots \\ = c_p (a_m x^m + b_n c_p ^n y^{pn} + b_n a_m c_p ^m y^n x^{pm} + \cdots)^p + \cdots .$$
Let us now look at the right-hand side:
$$(b_n y^n + \cdots) + \color{blue} {\big( a_m (f(x))^m + \cdots \big)} + (a_m x^m + \cdots) \color{green} {\big( b_n (f(y))^n + \cdots \big)} = \\ (b_n y^n + \cdots) + \color{blue} {\big( a_m (c_p x^p + \cdots)^m + \cdots \big)} + (a_m x^m + \cdots) \color{green} {\big( b_n (c_p y^p + \cdots)^n + \cdots \big)} = \\ b_n y^n + a_m c_p ^m x^{pm} + a_m b_n c_p ^n x^m y^{pn} + \cdots .$$
Let us agree to call "total degree" of a monomial $Ax^M y^N$ the number $M+N$. Let us agree to call "the total degree of a polynomial" the largest of the total degrees of its monomials.
Let us assume $m \le n$. Let us also assume $p > 1$ and try to obtain a contradiction.
In the left-hand side, the total degree of
$$a_m x^m + b_n c_p ^n y^{pn} + b_n a_m c_p ^m y^n x^{pm} + \cdots$$
is $n + pm$, therefore the total degree of
$$(a_m x^m + b_n c_p ^n y^{pn} + b_n a_m c_p ^m y^n x^{pm} + \cdots)^p + \cdots$$
is $pn + p^2 m$.
In the right-hand side, the total degree of
$$b_n y^n + a_m c_p ^m x^{pm} + a_m b_n c_p ^n x^m y^{pn} + \cdots$$
is $m + pn$.
Since both sides are equal, their total degrees as polynomials in $\Bbb R [x,y]$ must be equal, therefore $pn + p^2 m = m + pn$, whence follows $p^2 m = m$. There are two possibilities:
$m = 0$ - this is forbidden by the condition that $m$ be odd;
$m \ne 0$, whence it follows that $p^2 = 1$, so $p=1$, which is again impossible because we have assumed $p>1$.
It follows that our assumption was wrong, therefore $p \le 1$.
We cannot have $f=0$, because (replacing $f$ by $0$ in the original statement) we would get $Q=0$, which is not possible since the degree of $Q$ is odd (therefore it cannot be $-\infty$, unless you define $-\infty$ to be both odd and even).
We also cannot have $\deg f = 0$, because $\deg f$ is required to be odd.
It remains that $f(x) = cx + d$ with $c \ne 0$. Let us plug this back into the original statement and see what we get:
$$c P(x) + c Q(cy + d) + c Q(y) P(cx + d) + d = Q(y) + P(cx + d) + P(x) Q(cy + d) .$$
The leading term in the left-hand side is $c b_n y^n a_m c^m x^m$, coming from $c Q(y) P(cx + d)$. The leading term in the right-hand side is $a_m x^m b_n c^n y^n$, coming from $P(x) Q(cy + d)$. Equating them, we get $c^{m+1} a_m b_n = c^n a_m b_n$ and, since $c, a_m, b_n \ne 0$, it follows that $c^{n-m-1} = 1$. Since $n,m$ are odd, it follows that $n-m-1$ is odd, and the only real root of $c^{\color{blue} {\text{odd}}} = 1$ is $c=1$. Therefore, $f(x) = x + d$ and the original equality becomes
$$\tag{*} P(x) + Q(y + d) + Q(y) P(x + d) + d = Q(y) + P(x + d) + P(x) Q(y + d) .$$
Let us finally show that $d=0$. Begin by noticing that
$$(x+d)^i = \sum _{a=0} ^i \binom i a x^{i-a} d^a = x^i + d \sum _{a=1} ^i \binom i a x^{i-a} d^{a-1} ,$$
whence it follows that there exist polynomials $F_d$ and $G_d$ (with coefficients possibly depending on $d$) such that $P(x+d) = P(x) + d F_d (x)$ and $Q(x+d) = Q(x) + d G_d (x)$. This means that equality (*) can be rewritten as
$$P(x) + Q(y) + d G_d (y) + Q(y) P(x) + d Q(y) F_d (x) + d = Q(y) + P(x) + d F_d (x) + P(x) Q(y) + d P(x) G_d (y) ,$$
which, after simplifications, becomes
$$d \big( G_d (y) - Q(y) F_d (x) + 1 - F_d (x) - P(x) G_d (y) \big) = 0 .$$
There are two possibilities: the first one is $d=0$. The second one is
$$G_d (y) - Q(y) F_d (x) + 1 - F_d (x) - P(x) G_d (y) = 0 .$$
Let us show that this is in fact not possible. View the polynomial
$$\tag{**} G_d (y) - Q(y) F_d (x) + 1 - F_d (x) - P(x) G_d (y)$$
as a polynomial in $y$ with coefficients in $\Bbb R[x]$. As such, the leading term in $y$ is $-b_n y^n F_d (x)$, coming from the term $- Q(y) F_d (x)$. Since the polynomial is equal to $0$ and $b_n \ne 0$, it follows that $F_d = 0$, which means that equality (**) gets rewritten as
$$\tag{***} G_d (y) + 1 - P(x) G_d (y) = 0 .$$
Viewing this polynomial as a polynomial in $x$ with coefficients in $\Bbb R[y]$, its leading term is $-a_m x^m G_d (y)$, coming from the term $- P(x) G_d (y)$. Again, equality to $0$ and the fact that $a_m \ne 0$ imply that $G_d = 0$, whence (***) becomes $1 = 0$, clearly false. Therefore, $d=0$ is the only possibility.
We have proved, therefore, that $f(x) = x$ is the only solution. The only technique systematically used was looking at the leading terms of the various polynomials encountered.