I've already found a general formula to composite linear functions, i.e., if I have a function $f(x)=ax+b$, I can find a formula to $f^{(n)}(x)$ as function of the constants $a,b$ and $n$.
I want to know if it's possible to find a formula to the n-composite function of $g(x)=ax^2+bx+c$.
It is possible but computations get very cumbersome. I recommend starting with derivatives and then integrate the results.
for example let $$f(x)=g(g(x))$$
Then $$f'(x)=g'(g(x))g'(x)= (2ag(x) +b)(2ax+b) = 4a^2xg(x)+2ab (x+g(x))+b^2$$
Simplify and integrate and find the constant of integration by $x=0$.