Given the system:
$$ \left\{ \begin{array}{} \dot x=-x^3y^2 \\ \dot y = -2x^2y^3 \end{array} \right. $$
I need to find the equilibrium points and to determine whether the system is stable around them. I'v found $(0,0)$ to be a stable equilibrium point, using the Lyapunov function $V(x)=x^2+y^2$.
The rest of the equilibrium points are $(x_0,0), (0,y_0) \; , \; x_0,y_0 \in \Bbb R$. I'm having trouble with determining wether they are stable or not. Linearization is not useful in this case, and I couldn't find any Lyapunov function.
Here's a hint:
Your system is $$ \frac{d}{dt} \begin{pmatrix} x \\ y \end{pmatrix} = x^2 y^2 \begin{pmatrix} -x \\ -2y \end{pmatrix} , $$ which only differs from the linear system $$ \frac{d}{dt} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} -x \\ -2y \end{pmatrix} $$ by the nonnegative factor $x^2 y^2$.
This means that except for the $x$ and $y$ axes (where $x^2 y^2=0$), the trajectories for your system will be the same as for the linear system, up to a reparametrization of time.
Using this, you should be able to see what happens if you perturb an equilibrium $(x_0,0)$ or $(0,y_0)$ a little (so that you move off the axis).