For real numbers $a$ and $b \neq 0$, $f(x)$ satisfies the following:
$$f(f(x))=af(x)+bx$$
(1) $f(x)$ is continuous and $0<a, b<\frac{1}{2}$, show that the equation $f(x)=x$ has a real root, and find the real root.
(2) If $a>0>b, a^2+b \leq 0$, show that $f(x)$ is discontinuous.
I understood that $f$ is one to one, and if $f$ is continuous, it would be monotone.
Any hints??
(1): The sequence $x_n = f^{(n)}(x_0)$ satisfies $ x_{n+2} = a x_{n+1} + b x_n$. By induction, $$\lvert x_{2n} \rvert, \lvert x_{2n+1} \rvert \leq (a+b)^{n} max\{\lvert x_0 \rvert, \lvert f(x_0)\rvert\}$$ for all $n$. Therefore, $x_n \to 0$. By continuity of $f$ this implies $f(0)=0$. (vadim123's approach only shows that $0$ is the only candidate!)
(2): I think your monotonicity approach should work here. If $f$ is increasing then so are $f \circ f$ and $f^{(3)} = f \circ f \circ f$. But $$f^{(3)}(x) = (a^2+b) f(x) + abx,$$ so $f^{(3)}$ is decreasing (contradiction). If on the other hand $f$ is decreasing, then $f \circ f$ should be increasing, but $$ f\circ f (x) = a f(x) + bx $$ is decreasing (contradiction).