Proving that the origin is unstable in a dynamical system

Question

Prove that the origin is an unstable equilibrium point for the system $$\begin{align*}\dot{x}&=x^3+yx^2\\\dot{y}&=-y+x^2\end{align*}$$

I've already tried to linearize the system in order to use the Hartman–Grobman theorem, but the eigenvalues of the respective Jacobian matrix are $0$ and $-1$; non of them are positive, then Lyapunov's indirect method doesn't work either. Do you have any idea of another theorem that allow us to conclude this?

Answer 1

In order to determine that the origin of this dynamical system is unstable one can use Chetaev's instability theorem:

Let $V: D \to \mathbb{R}$ be a continuously differentiable function on a domain $D \subset \mathbb{R}^n$ such that $V(0)=0$ and $V(x_0)>0$ for some $x_0$ with arbitrarily small $\|x_0\|$. Choose $r > 0$ such that the ball $B_r = \left\{x \in \mathbb{R}^n\,|\,\|x\| \leq r\right\}$ is contained in $D$ and let $U = \left\{x \in B_r\,|\,V(x) > 0\right\}$ and suppose that $\dot{V}(x) > 0$ in $U$. Then, $x=0$ is unstable.

When evaluating the eigenvalues of the linearization at the origin the stable mode associated with eigenvalue of -1 is in the $y$-direction. Therefore, the following continuously differentiable function for in Chetaev's theorem is proposed

$$ V(x,y) = \frac12 (x^2 - y^2), \tag{1} $$

for which it can be shown that it has the following time derivative

$$ \dot{V}(x,y) = x^4 + (x - 1)\,y\,x^2 + y^2. \tag{2} $$

The set $\{z\in\mathbb{R}^2|V(z)>0\}$ using $(1)$ is equivalent to $|x|>|y|$, which does contain elements arbitrarily close to the origin.

When evaluating $(2)$ it can be shown that $\dot{V}(x,y)>0$ around the origin, for example the region $B_1$ (except the origin itself).

Combining all this yields that one can conclude that the origin of this dynamical system is unstable. Furthermore, it can also be stated that any trajectory that enters $U$ will leave $U$ at the boundary of $B_1$.

Answer 2

Determining the stable/unstable manifold $y=\phi(x)$ with $\phi(x) = \sum_{k=1}^n a_k x^k$

$$ \dot y = \phi'(x)\dot x $$

For $n = 3$ we obtain $y = \phi(x) = x^2$ hence $\dot x = x^3+\phi(x)x^2 = x^3+x^4$ which establishes an unstable flow at the origin.

In red, over the stream flow, the approximation $y = \phi(x)$