Find all functions $ f : \mathbb R \to \mathbb R $ such that for all $ a , b \in \mathbb R$: $$ f ( a ) + f \big( a + f ( b ) \big) = b + f \big( f ( a ) + f ^ 2 ( b ) \big) \text . $$
Here, for any $ n \in \mathbb N $, $ f ^ n $ denotes the $ n $-th iteration of $ f $.
My ideas so far:
I substituted $ ( 0 , x ) $ that yields: $$ f ( 0 ) + f ^ 2 ( x ) = x + f \big( f ( 0 ) + f ^ 2 ( x ) \big) \text . \tag 1 \label 1 $$
Let's say that $ a , b \in \mathbb R $, $ a \ne b $ and $ f ( a ) = f ( b ) $. Then the LHS does not change value with $ a , b $ but the RHS does. That is a contradiction and thus $ f ( a ) = f ( b ) \implies a = b $. The function is therefore injective.
If we substitute $ \big( x , f ( x ) \big) $ we can cross the functions like so: $$ f \big( x + f ^ 2 ( x ) \big) = f \big( f ( x ) + f ^ 3 ( x ) \big) \text ; $$ $$ \therefore \quad x + f ^ 2 ( x ) = f ( x ) + f ^ 3 ( x ) \text . \tag 2 \label 2 $$
If we assume $ f ( 0 ) = 0 $, we have $ f ^ 2 ( x ) = x + f ^ 3 ( x ) $.
Using \eqref{2} we get $ f ( x ) = 2 x $. However, this does not satisfy the functional equation and thus we can conclude that $ f ( 0 ) \ne 0 $.
I also noticed that if you substitute $ f ( x ) $ for $ x $, you get $ f ( x ) + f ^ 3 ( x ) =f ^ 2 ( x ) + f ^ 4 ( x ) $ and expressing $ f ^ 3 ( x ) $ from \eqref{2} we get an intresting result: $ f ^ 4 ( x ) = x $ which means the function is iterative with a cycle of $ 4 $ (or $ 2 $ or $ 1 $).
I am not sure how to continue or what substitution should I try next.
You can show that there is no function $ f : \mathbb A \to \mathbb A $ satisfying $$ f ( x ) + f \big( x + f ( y ) \big) = y + f \Big( f ( x ) + f \big( f ( y ) \big) \Big) \tag 0 \label 0 $$ for all $ x , y \in \mathbb A $, where $ ( \mathbb A , + ) $ is any abelian group with the neutral element $ 0 $ and the inverse function $ - $, such that there is $ b \in \mathbb A $ with $ 5 b \ne 0 $. As you are interested in $ \mathbb A = \mathbb R $ with the group operation $ + $ taken to be the usual addition of real numbers, this would be the case, since any nonzero real number can be chosen as $ b $.
To see this, substitute $ f ( x ) $ for $ x $, and see that $$ f \big( f ( x ) \big) - y = f \Big( f \big( f ( x ) \big) + f \big( f ( y ) \big) \Big) - f \big( f ( x ) + f ( y ) \big) \\ = f \Big( f \big( f ( y ) \big) + f \big( f ( x ) \big) \Big) - f \big( f ( y ) + f ( x ) \big) = f \big( f ( y ) \big) - x \text , $$ which in particular shows $$ f \big( f ( x ) \big) = f \big( f ( 0 ) \big) - x \text . \tag 1 \label{1a} $$ Putting $ x = 0 $ in \eqref{0} and using \eqref{1a} you have $$ f ( 0 ) + f \big( f ( 0 ) \big) - y = y + f \Big( f ( 0 ) + f \big( f ( 0 ) \big) - y \Big) \text , $$ which by letting $ a = f ( 0 ) + f \big( f ( 0 ) \big) $ and substituting $ - x + a $ for $ y $ shows that $$ f ( x ) = 2 x - a \text . \tag 2 \label{2a} $$ Using \eqref{1a} and \eqref{2a} you get $ 5 x = 0 $ for all $ x \in \mathbb A $, and in particular for $ x = b $, which is a contradiction.
In case every $ b \in \mathbb A $ is of order $ 5 $, choosing any $ a \in \mathbb A $ and taking $ f $ to be of the form \eqref{2a}, the equation \eqref{0} will be satisfied for all $ x , y \in \mathbb A $. To see this, use \eqref{2a} to rewrite \eqref{0} as $$ 2 x - a + 2 ( x + 2 y - a ) - a = y + 2 \big( 2 x - a + 2 ( 2 y - a ) - a \big) - a \text , $$ or equivalently $$ 4 x + 4 y - 4 a = 4 x + 9 y - 9 a \text , $$ which is true since $ 5 y = 0 $ and $ 5 a = 0 $. As the derivations resulting in \eqref{2a} were valid for any abelian group (independent of the order of its elemnts), we've characterized all the solutions it this case. Examples of abelian groups in which the order of every element is $ 5 $ are trivial groups, the cyclic group $ \mathbb Z _ 5 $ and the direct product of $ \mathbb Z _ 5 $ with itself.