The function $f:\mathbb{R}\rightarrow \mathbb{R}$ is said to have property (P) if $$ \text{(P) }f\text{ is continuous and }2f\left( f\left( x\right) \right) =3f\left( x\right) -x,\text{ }\forall x\in \mathbb{R}\text{.} $$
I want to prove that if $f$ has (P), then the set of all fixed points of $f$, $$ M=\left\{ x\in \mathbb{R}\text{, }f\left( x\right) =x\right\} $$ is a non-empty interval.
Denote $f^{n}:=\underset{n\text{ times }f}{\underbrace{f\circ ...\circ f}},$ $\forall n\geq 2,$ $f^{1}:=f.$ Since (P) one deduces inductively that $f$ fulfills the second-order recurrence $$ 2f^{n+2}\left( x\right) -3f^{n+1}\left( x\right) +f^{n}\left( x\right) =0,% \text{ }\forall x\in \mathbb{R},\text{ }\forall n\geq 1. $$ The characteristic equation $$ 2r^{2}-3r+1=0 $$ gives the roots $$ r_{1}=1\text{, }r_{2}=\frac{1}{2}. $$
Hence, $$ f^{n}\left( x\right) =a\left( x\right) +\frac{b\left( x\right) }{2^{n}},% \text{ }\forall x\in \mathbb{R}\text{, }\forall n\geq 1, $$ where $$ b\left( x\right) =-a\left( x\right) +x. $$
I don't know how to determine from now on the function $a\left( x\right) .$
I also want to find all functions having (P). Any help would be really appreciated.
Take any $a_0$. Define $a_1=f(a_0)$, $a_2=f(a_1)$, etc. Then $$ f(a_{n+1})-a_{n+1}=f(f(a_{n}))-f(a_{n})=\frac 12 (f(a_n)-a_n). $$ Thus $$ f(a_n)-a_n=\frac{1}{2^n}(f(a_0)-a_0). $$ Then $$ a_{N+1}-a_0= \sum_{n=0}^N a_{n+1}-a_n=\sum_{n=0}^N f(a_{n})-a_n=\sum_{n=0}^N \frac{1}{2^n}(f(a_0)-a_0)$$ Let $N\to\infty$, we see $a_{N+1}-a_0\to 2(f(a_0)-a_0)$. That is, $$ a_N\to c=a_0+2[f(a_0)-a_0]. $$ But $$f(c)=f(\lim a_N)=\lim f(a_N)=\lim a_{N+1}=c,$$ the second equal sign used $f$ being continuous. So we get a fixed point.
If $f$ has at least two fixed points, assume the fixed point set is not an interval. Then there exists an interval $[a, b]$ so that $f(a)=a$, $f(b)=b$ but $f(x)\neq x$ for $x\in (a, b)$. First assume $f(x)>x$ for $x\in (a, b)$. Take $a_0>a$ sufficiently close to $a$, so $f(a_0)-a_0$ is very small. Thus we get a fixed point $c=a_0+2[f(a_0)-a_0]$ that is very close to $a_0$, but bigger than $a_0$ and therefore inside $(a, b)$. Contradiction. If $f(x)<x$ for $x\in (a, b)$ we can choose $a_0<b$ very close to $b$ and similarly get a contradiction.
For classification, one can check $$ f=\left\{ \begin{array}{lllll} \frac x2+\frac a2 & \ \ \ \ \text{ if } x<a \\ x & \ \ \ \ \text{ if } a\leq x\leq b \\ \frac x2+\frac b2 & \ \ \ \ \text{ if } x>b \\ \end{array}\right. $$ Here $a\leq b$ are arbitrary. Sketch of proof: We see if $f(t)>t$ then there is a fixed point bigger than $t$; so after the fixed point interval $[a, b]$ we must have $f(t)<t$; similarly before the fixed point interval $[a, b]$ we must have $f(t)>t$. Observe if $x=a_0$ is not a fixed point, then all $a_n$ is not a fixed point, and if $f(a_0)>a_0$, then all $a_n$ satisfies $f(a_n)>a_0$. Similar for $<$. So for $a_0$ before the fixed point interval, it must be $a_n\to a$, i.e. $$ c=a_0+2[f(a_0)-a_0] \text{ must equal to } \,\, a $$
This implied $f(a_0)=\frac{a_0}{2}+\frac a2$.