About iterated functions I read here that for example for $f(x)=Cx+D$ we can calculate quite simply that:
$f^{[n]}(x)=C^nx+\dfrac{1-C^n}{1-C}D$
Likewise, I would like to obtain an explicit expression for the following function which is a fraction of two polynomials, and the denominator has a smaller order by one.
$f(x)=\dfrac{ax^4+bx^3+cx^2+dx+e}{a'x^3+b'x^2+c'x+d'}$
Is it possible to apply the same procedure for $f(x)$ here? Will that help if we assume $b,c,e, b', c'=0$? That is:
$f(x)=\dfrac{ax^4+dx}{a'x^3+d'}$.
If it is not possible to obtain an explicit expression for $f(x)$ Is there a known numeric algorithm through which I can quickly calculate $f^{[n]}(x)$ for large $n$?
One can try to proceed in a way similar to that of the Möbius transformations $$ f(x)=\frac{Ax+B}{Cx+D}. $$ For what follows, one must know the definition of projective space over $\mathbb R$ (https://en.wikipedia.org/wiki/Projective_space). In the second part of the answer, I try not to mention projective spaces.
So, let's consider a rational function of the form $$ f(x)=\frac{P(x)}{Q(x)}, $$ where $P,Q$ are polynomials of degree at most $k$. This function can be extended to the projective line $\mathbb P^1(\mathbb R)$ as follows: $$ F([x,y]):=[\tilde P(x,y),\tilde Q(x,y)], $$ where $\tilde P(x,y)=y^k P(x/y)$, $\tilde Q(x,y)=y^k Q(x/y)$. This is an extension of $f$ in the sense that $$ F([x,1])=[f(x),1]\quad \forall x\in\mathbb R \quad\text{s.t. $Q(x)\neq 0$}$$
From this, if we want to compute $f^{[n]}$, we can reduce ourselves to compute $F^{[n]}$. Namely, $$ F^{[n]}([x,1])=[f^{[n]}(x),1] \quad \forall x\in\mathbb R \quad\text{s.t. $Q(x)\neq 0$}.$$
In practice, what does it mean? (From here, one doesn't need to know the definition of projective space). Let's take your function $$f(x)=\dfrac{ax^4+dx}{a'x^3+d'}.$$ Let's define $\tilde P, \tilde Q$ and $F$ as above, namely (choosing $k=4$ as it is the maximum degree of the two polynomials): $$ F([x,y])=[{ax^4+dxy^3},{a'x^3y+d'y^4}].$$ Now, let's compute iteritevly $F^{[n]}$ (just considering it as a function from $\mathbb R^2$ to itself). Since we are composing polynomials, $F^{[n]}$ must be of the form $$ F^{[n]}([x,y])=[P_n(x,y),Q_n(x,y)], $$ with $P_n$ and $Q_n$ polynomials of degree at most $kn$. For instance, $$ P_2(x,y)=\tilde P(\tilde P(x,y),\tilde Q(x,y))=a(ax^4+dxy^3)^4+d(ax^4+dxy^3)(a'x^3y+d'y^4)^3.$$ Then, you will obtain $f^{[n]}$ simply taking $$ f^{[n]}(x)=\frac {P_n(x,1)}{Q_n(x,1)} $$
So, as long as you have an efficien way of composing polynomials (numerically for instance), you have a way of writing $f^{[n]}$. As I said in previous comments, though, the degree of $P_n$ and $Q_n$ might grow dramatically as $n$ grows, namely the degree will be at most $kn$, in your case $4n$.