I have tried for a long time to find a Lyapunov function for the specific problem
$$\begin{align} x' &= -x - 2y + xy^2 \\ y' &= 3x - 3y + y^3 \end{align}$$
Do you know any Lyapunov function what is going to work to determine the stability of the origin? According to a phase portrait produced by pplane the stability of the origin is asymptotically stable.
Thanks in advance!
Hint.
We have
$$ \cases{ 3x \dot x = -3 x^2-6 x y + 3x^2y^2\\ 2y \dot y = 6x y-6 y^2+2y^4 } $$
and after addition
$$ \frac 12\frac{d}{dt}(3x^2+2y^2) = -3x^2-6y^2+3x^2y^2+2y^4 $$
now there exist $\rho_m$ such that $x^2+y^2 < \rho_m^2$ makes
$$ -3x^2-6y^2+3x^2y^2+2y^4 < 0 $$
in fact, $\rho_m = 1$ suffices.