I have come to this from the topic of self-inverse functions. Let's consider a more complicated case where:
$$(f \circ f \circ f)(x)=x \tag{1}$$
Also assume that $f(x)$ is continuous on some non-empty interval, and $f^{-1}(x)$ exists on this interval.
Then it also follows:
$$(f \circ f)(x)=f^{-1} (x)$$
$$f(x)=(f^{-1} \circ f^{-1})(x)$$
$$x=(f^{-1} \circ f^{-1} \circ f^{-1})(x)$$
A trivial solution:
$$f(x)=x$$
And one non-trivial I know:
$$f(x)=x^{\exp (\pm 2 \pi i/3)}$$
Can we find any other non-trivial solutions? I would like both examples of particular solutions and the more general methods of finding them.
Update: dxiv offered another nontrivial solution (I guess $x \neq 0$ and $x \neq 1$):
$$f(x)=1-\frac{1}{x}$$
mercio provided a link to another related question: 3rd iterate of a continuous function equals identity function, which shows that if we require the function to be continuous and for the equation to hold on the whole real line, then only the trivial solution exists.
Which is why I do not require that, I'm fine with any interval $[a,b]$ with $a \neq b$.
Let $A$ be a non-singular $2\times2$ matrix and let $\lambda_1$ and $\lambda_2$ be 3rd roots of unity, and let
$$M = \begin{pmatrix}a&b\\c&d\end{pmatrix} :=A\begin{pmatrix}\lambda_1&0\\0&\lambda_2\end{pmatrix}A^{-1} \tag 1$$ Then the Linear Fractional Transformation
$$f(z) = \frac{az+b}{cz+d}$$ satisfies $f^3=id$. This is due to the correspondence between matrices and Möbius-Transformations, see for example this answer for more details.