I am trying to assess the (Liapunov) stability of the equilibrium at $(0,0)$ of the system \begin{align*} x_1' &= -4x_2 + x_1^2 \\ x_2' &= 4x_1 + x_2^2. \end{align*} I plotted the phase portrait in Mathematica, and it looks like a stable (but not asymptotically stable) equilibrium, with orbits circling about the origin. However, my professor tells me the origin is actually unstable. I don't know how to prove or disprove this rigorously. The equilibrium is non-hyperbolic, and I haven't been able to find a Liapunov function for the system. Any guidance would be appreciated.
Edit: I tried polar coordinates: I get $$ r' = \frac1r(x_1x_1' + x_2x_2') = r^2(\cos^3 \theta + \sin^3 \theta)$$ and $$ \theta' = \frac{x_1x_2'-x_1'x_2}{r^2} = 4 + r\cos\theta\sin\theta(\sin\theta-\cos\theta).$$ So there's a half-plane where $r$ is increasing and a half-plane where it's decreasing. The angle $\theta$ is increasing in a neighborhood of the origin. Is it reasonable to say that since $$ 0 = \int_{\theta_0}^{\theta_0 + 2\pi} \cos^3 \theta + \sin^3 \theta d\theta = \int_{\theta_0}^{\theta_0 + 2\pi} \frac{r'}{r^2} d\theta = \int_{\theta_0}^{\theta_0 + 2\pi} \left(\frac{1}{r}\right)' d\theta, $$ that the orbits are in fact periodic?

Dynamical system
$$ % \begin{align} % \dot{x} &= -4y + x^{2} \\ % \dot{y} &= 4x + y^{2} % \end{align} % $$
The sole critical point is the origin.
Phase portrait
The nullclines are the dashed lines. Red for $\dot{x} = 0$, purple for $\dot{y} = 0$.
Analysis
Polar coordinate transformation produces $$ \dot{r} = r^{2} \left( \cos^{3} \theta + \sin^{3} \theta \right) $$ The plot following shows $$ f(\theta) = \cos^{3} \theta + \sin^{3} \theta $$
The red zone shows contraction where $\color{red}{\dot{r} < 0}$, and the blue zone dilation, $\color{blue}{\dot{r} > 0}$
Notice that as we approach the origin, the dynamical system $$ % \begin{align} % \dot{x} &= -4y + x^{2} \qquad \Rightarrow \qquad -4y\\ % \dot{y} &= 4x + y^{2} \qquad \Rightarrow \qquad 4x % \end{align} % $$ approaches a periodic system.