$$\dot{x} = -2y -x^3 + x^2 y^2$$ $$\dot{y} = x - x^2 y$$
Jacobian evaluation at $(0, 0)$: $$\begin{pmatrix}0 & -2\\ 1 & 0\end{pmatrix}$$ Clearly the determinant is greater than zero but the trace is zero. Hence the linearization gives the impression of a center, which is marginally stable.
How do I show the origin is stable?
Related question: If in the case that the linearization of a system at a point is not marginally stable, is the stability of the linear system, the same as that of the non-linear system at the point (stable vs unstable)?
Near the origin we have
$$ \dot{x} = -2y + O(r^3)\\ \dot{y} = x +O(r^3) $$
here $r = \sqrt{x^2+y^2}$
The stream plot for the complete system
and for the approximate system
NOTE
Considering the approximated system we have
$$ x\dot x = -2x y\\ 2y\dot y = 2x y $$
adding
$$ \frac{d}{dt}(x^2+2y^2) = 0 $$
or
$$ x^2+2y^2 = C_0 $$
which means that the for the approximated system the origin is a center. [![enter image description here][3]][3]