This problem appears in In Perko section 2.9 problem 5b.
Use the appropriate Liapunov function to determine the stability of the equilibrium points of the following system: \begin{align} \frac{dx}{dt} &= x - 3y + x^3 \\ \frac{dy}{dt} &= -x + y - y^2 \\ \end{align}
My attempt at a solution for the equilibrium point $(x, y) = (0, 0)$:
If we linearize the system about the origin the we obtain a system with eigen values $\lambda = 1 \pm \sqrt{3}$, thus this is a hyperbolic unstable equilibrium point. So I would like $ \frac{dV(x(t), y(t))}{dt} > 0$ for my Liapunov function $V$.
Let $V(x, y) = ax^2 + by^2$ where $a, b > 0$. Then $$\frac{dV}{dt} = 2ax^2 + 2by^2 + 2ax^4 - 2by^3 + (-6a - 2b)xy.$$ This is close to what I want but the $xy$ term is causing me trouble. I've also tried miscellaneous functions of the form $V(x, y) = x^{2m} + y^{2n}$ but similar issues arise. What other forms of Liapunov functions can I test? Is it possible that there is a typo in the text?
The convention for the Liapunov function is $V(0, 0) = 0$ and $V(x, y) > 0$ for $(x, y) \neq (0, 0)$.
We can use the Chetaev's instability theorem:
$\bullet$ Let $x=0$ be an equilibrium point for the system $$ \dot x=f(x),\quad x\in D\subseteq\mathbb R. $$ Let $V:\,D\to \mathbb R$ be a continuously differentiable function such that $V(0)=0$ and $V(x_0)>0$ for some $x_0$ with arbitrarily small $\|x_0\|$. Suppose that there exists a neighborhood $B_r$ of the origin such that $$ \forall x\in B_r\quad V(x)>0\;\Rightarrow\;\dot V ( x ) > 0. $$ Then, $x=0$ is unstable. $\bullet$
Consider the function $$ V(x,y)=-xy - y^2; $$ $$ \dot V= -\dot x y-x\dot y-2y\dot y=-y(x - 3y + x^3)-x(-x + y - y^2)-2y(-x + y - y^2) $$ $$ =3y^2-x^3y+x^2+xy^2-2y^2+2y^3=y^2-x^3y+x^2+xy^2+2y^3 $$ $$ =x^2(1-xy)+y^2(1-x+2y) $$ From this point, it is obvious that the conditions of the theorem are satisfied ($\dot V$ is positive not only for $\{(x,y)\in B_r:\;-xy - y^2>0\}$, but for all $(x,y)$ in some neighborhood $B_r$ of the origin).