I have the following nonlinear system:
\begin{align}\dot x&=x(2-y^4) \\ \dot y &= -3x(1+y^3) \end{align}
I first verified this system using linearization; however, the eigenvalues are $\lambda =\pm i \sqrt{2} $ which is inconclusive for the stability of $(0,0)$. I tried to find a Lyapunov function to study its stability. I started with classical one $V(x,y) = a x^2 + b y^2$ by adjusting the coefficients $a,b$. However, it takes me hours but still no result. Any hints would be appreciated!! Thanks in advance.
It's obviously unstable; trajectories close to the eigendirection (2,-3) move away from the origin. Just look at the the phase portrait. And if that isn't enough to make you believe it, you can first consider the system $$ \dot x = 2-y^4 ,\qquad \dot y = -3(1+y^3) , $$ whose behaviour near the origin is very obvious (since now the origin is not an equilibrium), and then ponder what the effect of that extra factor of $\color{red}{x}$ in your system $$ \dot x = \color{red}{x}(2-y^4) ,\qquad \dot y = -3 \color{red}{x} (1+y^3) $$ is.