I have not seen any books about solving the equation of the following form: $af^2(x)+bf(x)+cx=0$ where $a$, $b$, $c$ are constants and $f^2(x)=f(f(x))$. We are going to find an expression of the function $f$. If I substitute $f(x)=kx$ in, I can get a solution, but I am interested in how to get ALL the solutions.
Please assume continuity if necessary.
Does any one know how to solve such functional equation?
The particular case of your equation
$$f^2(x)=x$$ is known as Babbage's functional equation.
If you have found a solution (say $f(x)=1-x$), you can build other ones using the following trick: let $\phi$ be an arbitrary invertible function, then let
$$g(x):=\phi^{-1}(f(\phi(x)))$$
With this definition, we have
$$g^2(x)=\phi^{-1}(f(\phi(\phi^{-1}(f(\phi(x))))))=\phi^{-1}(f(f(\phi(x))))=\phi^{-1}(\phi(x))=x.$$
So the space of solutions is huge.
I would expect the general equation to show similar richness, but I haven't found an easy way to adapt the trick.