I've finally resigned myself to believing that functions such as $f(x)=x^2+1$ have no iteration formulas in the closed form. I'm pretty sure that this is true, but I have no way of knowing for sure as of yet. Can anyone give me a way to determine and prove if a polynomial function can be iterated in the closed form?
I already know that functions of the form $f(x)=\frac{1}{a}(ax+c)^2-\frac{c}{a}$ can be iterated because they take the form $f(x)=(g\circ h\circ g^{-1})(x)$. Some other quadratics such as $f(x)=2x^2-1$ can be iterated using trigonometric functions. Maybe that helps.
Edit: Many quadratic functions "flip out" when you graph their iterates, and you get a strange and messy graph. These are usually the ones you can't iterate. The ones that can be iterated of the first form I mentioned have no inflection point, and those of the second type do, but each maximum and minimum has the same y-value. Just by messing around and graphing the iterates of a few parabolas with the line $y=x$ as a tangent at a single point, I noticed that their iterates also never have inflection points and don't behave erratically under iteration. It seems like they should be iteratable, but I can't figure out how.
As for quadratic polynomials, there are two special cases and only two, which have a finite iteration scheme:
1.) $\ ax^2+bx+ \frac{b^2-2b}{4a}$ iterates to $\frac{2A^{2^n} -b}{2a}$, where $A=\frac{2ax+b}{2}$;
2.) $ ax^2+bx+\frac{b^2-2b-8}{4a}$ iterates to $\frac {2B^{2^n}+2B^{-2^n}-b}{2a}$, where $B=\frac{2ax+b+\sqrt{(2ax+b)^2-16}}{4}$
Source: https://en.wikipedia.org/wiki/Iterated_function#Examples
As for higher-order polynomials, I cannot say. I have, however, found Carleman Matricies (https://en.wikipedia.org/wiki/Carleman_matrix) to be an interesting prospect provided you do not seek fractional iterations (that would require taking fractional powers of a matrix).
I hope you find this helpful.