unstable zero solution

Question

How can I choose the function $U$ , where $U(0)=0$ and $\dot U$ is positive definite for the system:

$$\dot x=y+xy^2$$ $$\dot y=x+yx^2$$

Because I want to show that the zero solution of the given system is unstable.

Answer 1

You'll get instability more easily by using the symmetry of the ODE. Namely, let $z=z(t)$ (scalar function) be the solution of $z'=z+z^3$ with $z(0)=\epsilon$. Clearly, $z$ increases with time and tends to infinity. Since $(z(t),z(t))$ is a trajectory of the given system with initial point $(\epsilon,\epsilon)$, it follows that $(0,0)$ is unstable.