I am supossed to study the instalibity of the null solution of the following (autonomous) system: \begin{equation*} \begin{cases} x' = -x+y-x^2 \\[5pt] y' =3x-x^2-2y \end{cases} \end{equation*} My resolution so far and my problems. Clearly we are working with an autonomous system with the form $z' = f(z)$, where $z' = [x'\hspace{.3cm} y']^T$ and $f(z) = f(x,y) = (-x+y-x^2,3x-x^2-2y)$. Obviously, $f \in C^1$ in $\mathbb{R^2}$ and $f(0,0)=(0,0)$ ($(0,0)$ is an equilibruim point). Let's compute the Jacobian of $f$. \begin{equation*} J_f(x,y) = \begin{pmatrix}-1-2x \quad \quad 1 \\[5pt] 3-2x \quad \quad -2 \end{pmatrix} \Rightarrow J_f(0,0) = \begin{pmatrix}-1 \quad \quad1 \\[5pt] 3 \quad \quad -2 \end{pmatrix} \end{equation*} Now we compute the eigenvalues of $J_f(0,0)$ and we get the following result: \begin{equation*} \lambda_1 = \frac{-3+\sqrt{13}}{2} \quad \vee \quad \lambda_2 = \frac{-3-\sqrt{13}}{2} \end{equation*} Which make us conclude that $J_f(0,0)$ isn't hurwitz (it has a positive real eigenvalue).
So, I can't conclude nothing about the stability of the null solution using this method. I would now try to find a Lupyanov function and use its direct method since these are the two methods I've been taught. I have tried multiple Lupyanov functions and I can't figure one out that actually works for this case. So this is my question, if someone can help me finding a Lupyanov function I would be really thankfull.
You got a strictly positive and a strictly negative eigenvalue. Thus the linearized system has a saddle point. This remains true in the local picture for the perturbation to the original non-linear system.
This kind of persistence of the characterization of the stationary point is only in doubt and can be violated if one or more of the eigenvalues is zero or has zero real part. See " Example of an ODE with an asymptotically stable equilibrium which is unstable in the corresponding linearization " for an example of how "unstable" (in a meta sense) this situation is.