Terminology for sets which are permuted within themselves on repeated application of a function

Question

Suppose that I have a function whose domain is equal to its codomain. One example of such a function is multiplication by a square matrix. In the matrix example, there often exist subspaces which are closed under repeated matrix multiplication - all eigenspaces have this property, as well as the plane perpendicular to the axis of a rotation in 3D. In the more general case where the function is nonlinear, what would be the correct name for these "closed subspaces," where repeated application of the function permutes the values within?

Answer 1

A more general answer is stable subsets. If the function is $f$ and the domain and codomain are $X$, then a subset $S$ of $X$ such that $f(S)\subset S$ is called $f$-stable. An orbit (in Kevin's sense) is $f$-stable, but not all $f$-stable subsets are orbits. Indeed, it is apparent from the definition that an orbit must be countable, so for example if $X$ is a real vector space, $f$ is a linear transformation, and $S$ is an eigenspace, then $S$ cannot be an orbit.

What is true in general is that an $f$-stable subset is a union of orbits.

A synonym for $f$-stable subsets is $f$-invariant subsets.

(Here, one needs to be a little careful with usage, because it's one thing for the set $S$ to satisfy $f(S)\subset S$ as a set, and another thing for the individual elements $x\in S$ to satisfy $f(x) = x$, and sometimes the word invariant is used with the latter case in mind. E.g. if the set $X$ being acted on is a ring, and $f$ is a ring automorphism, then the individual elements $x\in X$ satisfying $f(x) = x$ are called the $f$-invariants, and in this situation, the set of all $x$'s satisfying $f(x) = x$ is called the invariant subring. A set $S\subset X$ can be $f$-stable aka $f$-invariant even if the elements in it are not themselves invariants.)

Addendum: Accumulation's comment made me feel obligated to add that the condition $f(S)\subset S$ does not guarantee that $S$ is actually permuted by $f$, as the OP asked. Neither surjectivity nor injectivity are guaranteed. If $f$ is itself bijective on $X$, then its action on $S$ is injective, but even here it needn't be surjective. For example (as in the comments below), take $X=\mathbb{R}$, $S=[0,1]$, and $f(x) = 0.5x$. Then $S$ is $f$-stable, but $f$'s restriction to $S$ is not surjective.

In some situations, with adequate additional assumptions, $S$ being $f$-stable does guarantee that $S$ is permuted by $f$. For example, if $f$'s action on $X$ is bijective and $S$ is finite. Or, if $f$ has finite order, meaning that for some $n$, $f^n(x)=x$ for all $x\in X$. (In the latter situation, $f^n$ is the identity on $X$ and therefore on $S$, so it is bijective on $S$, and this forces $f$ to be bijective on $S$.) Or, if $X$ is a vector space of finite dimension and $S$ is a subspace and $f$ is a linear operator on $X$, and $f$ is nonsingular on $X$, then it is injective on $S$ so then it has to be surjective too, by counting dimensions.

(These are all conditions that are very frequently true in the situations I encounter as an invariant theorist, but e.g. an ergodic theory person would have a very different experience.)

For a general function $f:X\to X$, I do not know a one-word name for the condition on a subset $S\subset X$ that $f(S)\subset S$ and $f$'s restriction to $S$ is a permutation of $S$, which is what the OP actually asked. If I were to write about such a subset, I would write something like, "$S$ is $f$-stable and $f$'s restriction to $S$ is a bijection $S\to S$." Perhaps one could get away with "$S$ is $f$-stable and $f$ acts invertibly on $S$."

Answer 2

If we're talking about arbitrary functions, if $x$ is in your domain, then the set of values $\{x, f(x), f^2(x), f^3(x),\ldots\}$ is called the orbit of $x$.