A function $$F(n)=f(F(n+1))$$ is called $n$ times, from $n=b$ to $n=a$, where $b>a$, with the purpose of acquiring $F(a)$. Is there an elegant, mathematical way of depicting this?
To illustrate the iterations:
Loop 0: F(b)=0
Loop 1: F(b-1)=f(F(b))
Loop 2: F(b-2)=f(F(b-1))
...
Loop n: F(a)=f(F(a-1))
On
A fairly standard way of writing the $k$th iterate of a function $f$ is $f^k$. Because this is sometimes confused with the $k$th power of the function, especially when we write things like $$ \sin^2 x + \cos^2 x = 1, $$ where exponentiation really is what's intended, some folks write
$$f^{\circ k}$$
to indicate the $k$th iterate.
Read more here.
Mathematically, you were almost there:
\begin{align} F(n) &= f(F(n+1)) & n \in \mathbb Z, \ a \leq n < b, \\ F(b) &= 0. \end{align}
That's all a mathematician would need to see. The value of $F(a)$ is completely defined by this, although this requires the reader to actually think about how to produce the value of $F(a)$ given the value of $F(b).$
If you want to encode explicit step-by-step instructions, you can write the algorithm in the form of a computer program. If you want that to be a mathematical expression, you could try to learn denotational semantics, but that probably wouldn't actually make anything better.