I have a planer ODE system which is given by \begin{eqnarray} \begin{array}{lll}\begin{cases} \frac{dx}{dt} = p_{20}x^2+p_{11}xy+p_{30}x^3+p_{21}x^2y+p_{40}x^4+p_{31}x^3y+p_{22}x^2y^2,\\ \frac{dy}{dt} =-my+q_{11}xy+q_{21}x^2y+q_{31}x^3y, \end{cases}\end{array} \tag{1} \end{eqnarray} with $m,p_{20},p_{40},p_{22}$ and $q_{11}$ are all positive; $p_{11},p_{30}$ and $q_{21}$ are all negative; and the other coefficients $p_{21},p_{31}$ and $q_{31}$ can be any values. Clearly, the above system have an equilibrium point $E_0=(0,0)$. I have to find the stability (or, instability) properties of $E_0=(0,0)$ of the system $(1)$.
Now, using linearized stability analysis, the Jacobian matrix is given by $$J_{E_0}=\begin{pmatrix} 0 & 0\\ 0 &-m \end{pmatrix}.$$ Clearly, the eigenvalues of $J_{E_0}$ are $0$ and $-m$. So, the linearized stability analysis fails to give the local stability properties of $E_0=(0,0)$.
Now, we will move towards the center manifold theory to conclude the stability properties of $E_0$.
According to the Theorem $18.1.2$ given in Wiggins dynamical system book, page-246, there exists a center manifold for $(1)$ which can be locally represented as
$$W^c(0) =\{(x,y)\in \mathbb{R}^2|~~y=h(x), |x|<\delta, h(0)=Dh(0)=0\},$$
where $D$ denotes the derivative and $\delta$ is sufficiently small. Then we consider $h(x)$ as \begin{equation} h(x) = ax^2 + bx^3 +cx^4+ O(x^5).\tag{2} \end{equation} According to $(18.1.10)$ from Wiggins same book, page-248, the equation for the center manifold is given by \begin{equation}\mathcal{N}(h(x))=Dh(x)[Ax+f(x,h(x))]-Bh(x)-g(x,h(x))=0,\tag{3} \end{equation} where $A=0$, $B=-m$, $f(x,y)=p_{20}x^2+p_{11}xy+p_{30}x^3+p_{21}x^2y+p_{40}x^4+p_{31}x^3y+p_{22}x^2y^2$, and $g(x,y)=q_{11}xy+q_{21}x^2y+q_{31}x^3y$.
Now we obtain the value $a$, $b$ and $c$ in $(2)$, by calculating the coefficient of $x^i$ in $(3)$.
Coefficient of $x^i$:
$\qquad x^2:\quad ma=0\implies a=0$
$\qquad x^3:\quad 2p_{20}a+mb-q_{11}a=0\implies b=0$
$\qquad x^4:\quad 2p_{11}a^2+3p_{20}b+mc-q_{21}a=0\implies c=0$
$\qquad\vdots$
So, $h(x)=0$. Thus the function which determines the center manifold becomes zero. Here, I am stuck.
Any help will be highly appreciated. Also is there any other method/approach available for the stability of $E_0$?
Follows a MATHEMATICA script which calculates the leaves of the center manifold. The script is self explanatory. We can see that in this case we have one leave, being the $x$ axis.