Let $a$ and $b$ be real numbers in the interval $(0,1/2)$ and let $g$ be a continuous real-valued function so that $g(g(x)) = ag(x)+bx$ for all real x. Prove that $g(x)=cx$ for some constant c.
Let $x_0$ be a real number. One can show that g is injective easily. To see why g is surjective, observe that $g$ is either strictly increasing or strictly decreasing by continuity and injectivity. g cannot tend to a finite limit as n tends to infinity since otherwise $g(g(x))-ag(x)=bx$ would be bounded and similarly g cannot tend to a finite limit as $x\to -\infty$. By the IVT and monotonicity, g is surjective. Now we can define the sequence $x_n$ by $x_n = g(x_{n-1})$ for $n\ge 1$ and $x_{n-1} = g^{-1}(x_n)$ for $n\leq 0$. One can then observe that the roots of the characteristic equation of this sequence are $r_1,r_2=(a\pm \sqrt{a^2+4b})/2$. So $x_n = c_1r_1^n + c_2 r_2^n$ for all $n\in\mathbb{Z}$.
I get that the formula holds for nonnegative n, but for $n > 0,x_{-n} - a x_{-n-1}-bx_{-n-2} = 0,$ so wouldn't the characteristic equation for $(x_{-n})_{n>0}$ be $bx^2 +ax-1=0$? This seems to lead to a different formula for $x_n$.
Edit: Note that if $x$ is a root of $x^2-ax+b,$ then $b\neq 0\Rightarrow x\neq 0$ and so $1/x$ is a root of $bx^2+ax-1.$ Conversely, if $x$ is a root of $bx^2 + ax-1$, then $1/x$ is a root of $x^2-ax+b$. This generalizes to arbitrary polynomials when one reverses the coefficients. For $n\ge -1$, $x_{-n} = c_1' r_1^{-n} + c_2' r_2^{-n}$ for some constants $c_1'$ and $c_2'$. Then the values for $x_0$ and $x_1$ show that the coefficients for the roots of $x_{-n}$ are indeed $c_1$ and $c_2$.
But how can one analyze this to conclude that $g$ has the required form? I was thinking of taking limits and using the facts that $r_1 > 0>r_2, 1 >|r_1|>|r_2|$.