Question 1: Study the existence of $C^1$ function $f : \mathbb{R} \rightarrow \mathbb{R}$ satisfying $\forall x\in\mathbb{R},\mbox{ } f\circ f'(x)=x.$
Question 2: Study the existence of differentiable function $f : \mathbb{R} \rightarrow \mathbb{R}$ satisfing $\forall x\in\mathbb{R},\mbox{ } f\circ f'(x)=x.$
Question 3: Study the existence of $C^1$ function $f : \mathbb{R} \rightarrow \mathbb{R}$ satisfying $\forall x\in\mathbb{R},\mbox{ } f'\circ f(x)=x.$
For question 1, such a function can not exist because : f' must be injective and since f 'is continuous, f' must be strictly monotonous. For example, If we assume that f 'is strictly increasing We can show that $\displaystyle \lim_{x\rightarrow -\infty}f'(x)=-\infty$ and $\displaystyle \lim_{x\rightarrow +\infty}f'(x)=+\infty$ which implies that f ' is surjective .
with a simple argument it shows that f is injective ( if $f(x)=f(y)$ by surjection of $ f'$ , we have $f'(a)=x $ and $f'(b)=y$ for some real $ a,b $ , thus implie $a=f(f'(a)=f(x)=f(y)=f(f'(b)=b$ so $ x=y$).
the continuity of f proves that f est strictly monotonous. For example, if we suppose f strictly inreasing, we must have $f'>0$. this contradicts the surjectivity of $f '$
For Question 2 , I need help
Problem 1. We show that no such $f$ exists.
It is clear that $f$ is surjective and $f'$ is injective. Since $f'$ is continuous, $f'$ is strictly monotone.
Now we show that $f'$ is surjective. We know that $f'(\mathbb{R}) = (\alpha, \beta)$ for some $-\infty \leq \alpha < \beta \leq +\infty$. Then we have to prove that $\alpha = -\infty$ and $\beta = +\infty$. If both $\alpha$ and $\beta$ are finite, then $(f\circ f')(\mathbb{R}) \subseteq f([\alpha, \beta])$, which contradicts the surjectivity of $f$. So it follows that either $\alpha = -\infty$ or $\beta = +\infty$.
Other cases can be treated in a similar way to yield the contradiction, hence we conclude that $f'$ is surjective.
Now since $f'$ is bijective, the functional equation tells that $f$ is also bijective and is the inverse function of $f'$. But this implies that $f$ is monotone, which then contradicts the surjectivity of $f'$.
Problem 2. Again we show that no such $f$ exists.
We make use of the previous problem. To this end, it suffices to show that the given condition implies the continuity of $f'$. The following general lemma will be sufficient for our purpose:
Proof. By the Darboux theorem, we know that $g'$ possesses intermediate-value property. Together with the injectivity of $g'$, this implies that $g'$ is strictly monotone. In particular, $g'$ can only have jump-discontinuities. Then again by the intermediate-value property, $g'$ cannot have jump-discontinuities. So $g'$ is continuous everywhere.
Problem 3. Again no such function exists.
Assume that $f$ solve the functional equation. Then $f$ is injective and $f'$ is surjective. The former implies that $f$ is strictly monotone, which in turn tells that $f'$ is either non-negative or non-positive. But this contradicts the surjectivity of $f'$.
Addendum. Interestingly, there exists a $C^1$-function $f$ such that $(f' \circ f)(x) = |x|$. Indeed, we can make an ansatz that $f$ takes the form $f(x) = c |x|^{\alpha}$ for some $\alpha > 1$ and $c > 0$ and then plug this to the equation to determine $\alpha$ and $c$. Then we should have $c^{\alpha}\alpha|x|^{\alpha^2 - \alpha} = |x|$, and so, we may choose $\alpha$ as the golden ratio $\phi = \frac{1+\sqrt{5}}{2}$ and then $c = \phi^{-1/\phi}$.