I have a function $f(x)$ defined on a domain $D$, but such that the image $f(D)$ may contain extra regions not included in its domain. I am interested in solving the fixed-point equation $x=f(x)$. If I do a fixed-point iteration:
$$x_{n+1} = f(x_n)$$
starting from a point $x_0\in D$, I risk that some point $x_n\notin D$ falls outside the domain.
Are there techniques to deal with such an $f$? For simplicity let's assume that $f$ is smooth in $D$.
I am aware of one such situation (of course there may be more—I only know of this one though): Browder’s demiclosedness principle. This states that if you have a closed, bounded, convex subset $C$ of a uniformly convex Banach space $X$, then any nonexpansive mapping $f : C \to X$ is such that $I-f$ is demiclosed at 0.
That is, we say a function $T$ is demiclosed at $y$ if $x_n$ converges weakly to $x$ and $T(x_n)$ converges strongly to $y$, it follows that $Tx=y$. In particular this tells you that $f$ will have a fixed point if $f$ admits an approximate fixed point sequence (which will occur if $f(C) \subseteq C$, for instance, which you explicitly say is not the case in your situation).
Here is a primary reference for this work if this at all resembles the kind of situation you might be interested in: “Semicontractive and semiaccretive nonlinear mappings in Banach spaces,” Bulletin of the AMS, by Felix Browder.