I'm here again busy determining stability of a stationary point of ordinary differential equation. I've asked here to get help with determining the stability and I wanted to avoid center manifold theorem, but a good stackexchange user suggested me to do it, so I have done it. However, it seems that it is not easy to learn the whole manifold theory in just two days or something. What I have focused on is the reduction principle. But there are still things that I don't understand and the most books and articles that are online do not explain it. After reducing the differential equation till something like: \begin{align} x'= -x^3 + O(\Vert(x,y)\Vert^4), \hspace{10pt} y'=-y^3 + O(\Vert (x,y)\Vert^4) \end{align} They (without any explanation) say that the negative coefficient $(-1)$ implies stability and that the high order terms do not affect that. Okay I can understand it a little bit, but I would like to see a proof that this is actually true.
In my case I have got something less trivial than that after putting my system in a different coordinate for which the reduction principle can be applied and then reducing it. I've got: \begin{align} x'=&\alpha y + \beta x y + O(\Vert(x,y)\Vert^3)\\ y'=&-\alpha x + \gamma x y + O(\Vert(x,y)\Vert^3) \tag{1} \end{align} where the coefficient are positive. If I negelect the high order terms and just apply Lyapunov's second method on: \begin{align} x'=&\alpha y + \beta x y\\ y'=&-\alpha x + \gamma x y \end{align} Using the following Lyapunov function: \begin{align} V(x,y)=\frac{1}{\beta} x - \frac{\alpha}{\beta^2}\ln(\alpha+\beta x) - \frac{1}{\gamma}y-\frac{\alpha}{\gamma ^2}\ln(\alpha - \gamma y) + \alpha \left(\frac{1}{\beta ^2 } +\frac{1}{\gamma ^2} \right)\ln(\alpha) \end{align}
I see that it is stable, because $V'=0$. Now my question is:
Question: Can I neglect the high order terms in (1)? If yes, what is the proof for it?
I would like to avoid proofs that need deep manifold theory in it. Can you also prove it without knowing anything about manifold theory, please? Furthermore, I don't need a fully written proof for it, just a sketch would be fine, so I fill in the details myself. Thanks in advance.
For your particular case, the answer is definitely no, you cannot ignore the higher order terms. They are actually the most important predictors of the stability of the system. Your system is \begin{align} \dot{x} &= \alpha \, y + \beta\, xy + f(x,y)\\ \dot{y} &= - \alpha \, x + \gamma\, xy + g(x,y) \end{align} where $f(x,y)$ and $g(x,y)$ are functions of order three and higher. The truncated system \begin{align} \dot{x} &= \alpha \, y + \beta\, xy \\ \dot{y} &= - \alpha \, x + \gamma\, xy \end{align} is actually integrable (conservative) which means that it has a conserved quantity (i.e. a first integral). That first integral is the Lyapunov function you have written down. For that reason, the orbits of the truncated system are closed ovals, following the level curves of the first integral (the Lyapunov function). Integrability implies structural instability. That is, the perturbing residual terms $f(x,y)$ and $g(x,y)$ generically destroy the conservative motion of the truncated system and the stability could go either way - you may end up having either a stable equilibrium point or an unstable one.
Bottom line: the expansion you have obtained so far does not give you any information as it is.