Suppose we have a non-linear autonomous system of two equations:
$$\begin{cases} x'(t) = F(x,y) \\ y'(t) = G(x,y) \end{cases} $$
and we obtain a fixed point for this equation, but the eigenvalues of the Jacobian matrix evaluated at the fixed point are equal to zero.
Then how would one proceed to check the stability? Does this happen? could I have an example or a reference.
If an eigenvalue has no imaginary part and is equal to zero, the system will be unstable, since, a system will not be stable if its eigenvalues have any non-negative real parts. This is just a trivial case of the complex eigenvalue that has a zero part.