Linearization of $C^r$ ODE system

Question

Consider the following ODE system \begin{align*} \dot{x} &= x^5 + y^3 = f(x\,,y) \\ \dot{y} &= x^3 - y^5 = g(x\,,y) \end{align*} Clearly, $(x\,,y) = (0\,,0)$ is the only fixed point. Linearizing $$ J = \begin{bmatrix} 5x^4 & 3y^2 \\ 3x^2 & -5y^4 \end{bmatrix} $$ evaluating at the fixed point will just give a zero matrix. Thus, need to use some other approach to analyze the stability, e.g., Lyapunov function.

However, I can't find any Lyapunov function related to this problem. Since this ODE system is at least $C^3$ in both $x$ and $y$. So I am wondering, for the linearization, is it possible to go all the way to 3rd-order derivative? Maybe something like $$ \begin{bmatrix} \frac{\partial^3f}{\partial x^3} & \frac{\partial^3f}{\partial y^3} \\ \frac{\partial^3g}{\partial x^3} & \frac{\partial^3g}{\partial y^3} \end{bmatrix} $$ This way the matrix will not be a zero matrix and can find corresponding eigenvalues and eigenvectors.

Answer 1

From

\begin{align*} \dot{x} &= x^5 + y^3 \\ \dot{y} &= x^3 - y^5 \end{align*}

we have

\begin{align*} x^3\dot{x} &= x^8 + x^3 y^3 \\ y^3\dot{y} &= x^3y^3 - y^8 \end{align*}

and then

$$ \frac 14\frac{d}{dt}(x^4-y^4) = x^8+y^8 $$

analyzing the behavior along a line $y(t) = \alpha x(t)$ we conclude

$$ \frac 14(1-\alpha^4)\frac{d}{dt}x^4 = (1+\alpha^8)x^8 $$

so depending on $\alpha$ the dynamics along $y(t) = \alpha x(t)$ changes concluding that the origin is unstable.

Answer 2

Looking for the central manifold, considering the system

$$ \cases{ \dot x = f(x,y) = x^5+y^3\\ \dot y = g(x,y) = x^3-y^5 } $$

and considering $y = h(x) = \sum_{k=0}^n a_kx^{k+1}$ we have

$$ h'(x)\dot x = \dot y\Rightarrow h'(x)f(x,h(h))=g(x,h(x)) $$

for $n = 6$ we get two solutions:

$$ h_1(x) = -\frac{2 x^7}{27}-\frac{x^5}{6}-\frac{x^3}{3}-x + O(x^9)\\ h_2(x) = -\frac{2 x^7}{27}+\frac{x^5}{6}-\frac{x^3}{3}+x + O(x^9) $$

Attached a stream flow plot showing the phase plane in light blue and $h_1(x), h_2(x)$ in red. As can easily be verified, $y=h_1(x)$ has stable flow and $h_2(x)$ has unstable flow.