Find all real numbers $a,b,c\in\mathbb{R}$ for which there exists a function $f:\mathbb{R}\to\mathbb{R}$ such that: $$ f(f(x))=ax^2+bx+c $$ for all $x\in\mathbb{R}$.
The only thing I could deduce is: $$ f(ax^2+bx+c)=af(x)^2+bf(x)+c $$ Which doesn't help much. How to tackle the problem?
This is only a partial answer, but it might be of some help.
If $f(x)=mx+d$, then $f(f(x))=m(mx+d)+d=m^2x+(m+1)d$, so any triple of the form $(0,b,c)$ with $b\ge0$ works, by taking $m=\sqrt b$ and $d=c/(1+\sqrt b)$.
Likewise, if $f(x)=m|x|^\sqrt2$, then $f(f(x))=m|(m|x|^\sqrt2)|^\sqrt2=m|m|^\sqrt2x^2$, so any triple of the form $(a,0,0)$ works, by taking $m=sgn(a)|a|^{1/(1+\sqrt2)}$.
So it looks to me like there are two natural questions: 1) Does $a\not=0$ force $b=c=0$? and 2) are there any triples with $b\lt0$?