Let $f:A\rightarrow A$ be a function. We can simply define $f\circ f$, $f\circ f\circ f$, etc., for each given natural number inductively.
$f^{(0)}=id_A$
$\forall n\in\omega \qquad f^{(n+1)}=f\circ f^{(n)}$
How to define $f^{(\omega)}$ such that it be a function from $A$ to $A$? Is it possible to define $f^{(\alpha)}$ for arbitrary large ordinal number $\alpha$?
Yes, and no. Essentially, sometimes.
Consider the function $f(x)=-x$ for $x\in\Bbb R$. Then what would $f^{(\omega)}(x)$ be? It should be a limit of $f^{(n)}(x)$. But there is no limit.
Or consider the function $f(x)=x+1$ for $x\in\Bbb N$. Then $f^{(\omega)}(0)$ is what?
But consider the function $f(x)=x+1$ for $x\in\omega_1$, then $f^{(\omega)}(x)=x+\omega$ which is perfectly defined. But then $f^{(\omega_1)}$ causes the same issues as before.
You can sometimes talk about infinite iterations, but you need to have some mechanism to handle the limit steps. This usually requires topology, and convergence. And even then you may have to end up changing the domain a little bit.
So under additional assumptions (e.g. the function is surjective, and there is a way to assign values at limit points), yes. In general no.