How do I notate the set or group created by iterated composition of functions? Is there a convention?
In particular I have a function $f(x)$ and I want to notate the following set:
$F=\{f^n(x):n\in\mathbb{N}\}$
where $f^2(x)=f(f(x))$ etc.
Is there a convention? I want to write:
$F=f^{\mathbb{N}}(x)$ ?
which I'm sure is clear and everybody would understand, but when I invent notation, people get cross (although they always seem to understand). So I thought I better ask first.
On
If you consider $F := \{f^n|n \in \mathbb{N}\}$, you will not have a group, as not all functions are bijections, and therefore they are not invertible.
Note that you should specify what domain and codomain $f$ must have so we can know whether it makes sense to consider function composition. As it stands there, I assume domain and codomain are the same, otherwise it would be impossible to iterate function composition.
Considering your notation: I would just leave it as I did in the first sentence of this answer. This will usually be well understood, certainly if you add context where you explain that you use function composition..
A small remark:
If we consider the set $\{f: X\rightarrow X | f$ bijective$\}$, we do have a group, and this group is usually denoted as $Sym(X)$. Or if $X = \{1,2, \dots, n\}$, we also write $S_n$.
Another small remark:
$f(x)$ is not a function, it is a function $f$ that is applied to an element $x$ in the domain.
There may not be a conventional notation, as others have commented, but there is a conventional terminology. Also, there is a conventional way to bring groups into the picture, although not in the way that you have notated.
The function $n \mapsto f^n$ is called an action of $\mathbb{N}$, and the set $\{f^n(x) \,|\, n \in \mathbb{N}\}$ is called the orbit of the point $x$ under this action.
I see you did not indicate the domain of the function $f$; if I bring the domain into the picture with notation $f:X \to X$, then the function $n \mapsto f^n$ is called an action of $\mathbb{N}$ on $X$.
So far, I haven't mentioned groups. I agree with the comments and answers that it is inappropriate to refer to $\{f^n(x) \,|\, n \in \mathbb{N}\}$ as a group.
However, there is a way to bring group theory into the discussion.
First, let's assume that $f$ is a bijection. In that case $f$ is an element of the set of permutations of the set $X$, which I'll denote $Sym(X)$. The set $Sym(X)$ is a group under the operation of function composition: for all $f,g \in Sym(X)$ we have $f \circ g \in Sym(X)$; the associative law holds $(f \circ g) \circ h = f \circ (g \circ h)$; and because each element $f \in Sym(X)$ is a bijection, it has an inverse $f^{-1}$ satisfying the inverse law $f^{-1} \circ f = f \circ f^{-1} = \text{Id}_X$.
So we have $f \in Sym(X)$.
Second, let's define the function $\mathcal{A} : \mathbb{Z} \to Sym(X)$ by the formula $$\mathcal{A}(n) = f^n $$ Then we can easily derive the formula $f^n \circ f^m = f^{n+m}$. In other words, $\mathcal{A}$ is a homomorphism from the group $\mathbb{Z}$ to the group $Sym(X)$. The image of this homomorphism is the subset $$\{f^n \,|\, n \in \mathbb{Z}\} \subset Sym(X) $$ And now we can bring in a standard result from group theory, which says that the image of every homomorphism from one group to a second group is a subgroup of the second group.
In conclusion, as long as $f$ is assumed to be a bijection, it follows that the set $\{f^n \,|\, n \in \mathbb{Z}\}$ does indeed form a group under the operation of composition.
Note well, however, that this does not mean that the set $\{f^n(x) \,|\, n \in \mathbb{Z}\}$ is a group, nor $\{f^n(x) \,|\, n \in \mathbb{N}\}$ as originally asked. Those sets are more properly called orbits of the action (of $\mathbb{Z}$ or of $\mathbb{N}$, respectively).