Let $F:\mathbb{R}^2\to\mathbb{R}^2$ be $C^\infty$. Let $(x_0,y_0)$ be a fixed point of $F$. My question is as follows, how would I go about figuring out if $(x_0,y_0)$ is an isolated fixed point or if there are a continuum of fixed points near by. To provide a rigorous definition of the latter I would say something along the following (but am not wedded to this definition)
There exists $\epsilon>0$ and $h(z)$ which is continuous in $z$ and $h(z)=0$ such that for all $z$ with $|z|<\epsilon$ we have $F(x_0+z,y_0+h(z))=(x_0+z,y_0+h(z))$
Essentially, when looking at the displacement map $F(x)-x$, I would like to be able to distinguish between figure 3.1 and the first zero in 3.2 on page 805 of this paper, or found here:
I know that it is necessary that $(x_0,y_0)$ is a critical point of $F(x)-x$ but that does not distinguish between the two cases Im interested in. Is there a quick/straight forward way to check?
Pose $G(x,y)=F(x,y)-(x,y)$. Then fixed points of $F$ are zeroes of $G$.
If $(x_0,y_0)$ is a zero of $G$ and $G$ is $\mathbb{R}^2$-differentiable around $(x_0,y_0)$, then posing $(h,k)=(x-x_0,y-y_0)$, $G(x,y)$ is locally equivalent to:
$\begin{cases} G_x(x,y) = \frac {\partial G_x} {\partial x} h + \frac {\partial G_x} {\partial y} k \\ G_y(x,y) = \frac {\partial G_y} {\partial x} h + \frac {\partial G_y} {\partial y} k \end{cases}$
This system admits only $(h,k)=(0,0)$ as zero for $G(x,y)$, except if
$\frac {\partial G_x} {\partial x} \frac {\partial G_y} {\partial y} - \frac {\partial G_x} {\partial y} \frac {\partial G_y} {\partial x} = 0$
So a sufficient condition to have an isolated zero for $G$ on $(x_0,y_0)$ is
$\frac {\partial G_x} {\partial x} \frac {\partial G_y} {\partial y} - \frac {\partial G_x} {\partial y} \frac {\partial G_y} {\partial x} \ne 0$.
Coming back to $F$, a sufficient condition for $(x_0,y_0)$ to be an isolated fixed point is
$(\frac {\partial F_x} {\partial x} - 1) (\frac {\partial F_y} {\partial y} - 1) - \frac {\partial F_x} {\partial y} \frac {\partial F_y} {\partial x} \ne 0$
(with all partials computed on $(x_0,y_0)$).
OP remarked in comments that this is not sufficient to distinguish between cases 3.1 and 3.2, which was the main request. So we have to go one level deeper in partial derivatives. For simplification we'll use now vectors, with $DG_x = (\frac {\partial G_x} {\partial x}, \frac {\partial G_x} {\partial y})$, and $v=(h,k)$.
We are still on the point $(x_0,y_0)$ which is a zero of $G$. At order 2, we get:
$\begin{cases} G_x(x,y) = DG_x v + \frac 1 2 v^T HG_x v \\ G_y(x,y) = DG_y v + \frac 1 2 v^T HG_y v \end{cases}$
where $H$ is the Hessian operator, i.e.
$HG_x = \begin{pmatrix} \frac {\partial^2 G_x} {\partial x^2} \frac {\partial^2 G_x} {\partial x \partial y} \\ \frac {\partial^2 G_x} {\partial y \partial x} \frac {\partial^2 G_x} {\partial y^2} \end{pmatrix}$
The condition $\frac {\partial G_x} {\partial x} \frac {\partial G_y} {\partial y} - \frac {\partial G_x} {\partial y} \frac {\partial G_y} {\partial x} = 0$, which is verified by both 3.1 and 3.2 cases, means that matrix $DG$ has a null determinant, where
$DG=\begin{pmatrix} \frac {\partial G_x} {\partial x} \frac {\partial G_x} {\partial y} \\ \frac {\partial G_y} {\partial x} \frac {\partial G_y} {\partial y} \end{pmatrix}$
If $|DG|=0$, $DG$'s kernel can have dimension 1 or 2. It is among this kernel that we want to know the effect of $HG$:
To continue exploring this last case, we might use order-3 partial derivatives. However, whatever the order of derivatives we use, there are cases that cannot be settled, such as (for $\mathbb{R} \to \mathbb{R}$ functions, but it can be adapted) $\exp(-\frac 1 {x^2})$ whose derivatives on $0$ are all null.