I am studying the book by Holmes and Guckenheimer, and am relatively new to all this Lyapunov functions, so I am trying to do all the exercises in it. I am currently stuck with exercise 1.3.1. It is about the equation $x'' + \epsilon x^2 x' + x = 0$, with $\epsilon > 0$, which has the particularity that when linearized the eigenvalues have null real part, so we cannot apply the theorem that says there is an homeomorphism between flows. The exercise asks to prove via a Lyapunov function that the point $x=0, x'=0$ is assimptotically stable.
My first guess was a very simple function $V(x,y) = \frac{x^2}{2} + \frac{y^2}{2}$ where $y=x'$, and I got $V'(x,y) = -\epsilon x^2y^2$ which is not enough (the derivative is along the curves) to prove assimptotic stability. Then I tried something along the lines of a previous example in the book, $V(x,y) = \frac{x^2}{2} + \alpha \frac{y^2}{2} + \gamma xy$, but now $V' = xy(\cdots) + \gamma(y^2 - x^2)$ so I cannot get any stability from here.
Any advice will be very appreciated.
You clearly need $\epsilon>0$ for stability of the origin.
Hence $V' =0$ only when $x=0$ or $y=0$. Neither $x \equiv 0$ or $y \equiv 0$ are invariant except at the origin. Hence, by La Salle's theorem, origin is globally asymptotically stable.
Note: This is to answer the question in OP's comment.
I am not sure you can do without La Salle's theorem as it is the only general tool we have when $V'$ is only semi-definite.
Going to the original problem, the equations are $$\begin{align} x' &= y \\ y'&= -x - \epsilon x^2 y \end{align}$$ If we let $$V(x,y)=\frac{x^2+y^2}{2}$$ then along any trajectory $$V' = x x' + y y' = -\epsilon x^2 y^2 \le 0$$
Now $V$ decreases except when $x=0$ or $y=0$ or both.
1) Suppose at some point in time, $x =0$ and $y \neq 0$. Then $x' \neq 0$, and $x$ will have to change and become $\neq 0$.
2) Suppose at some point in time, $x \neq 0$ and $y=0$. Then $y' \neq 0$ and $y$ will have to change and become $\neq 0$.
Thus the trajectory cannot stay on $x=0$ or $y=0$ unless both are zero, i.e. at the equilibrium. Thus $V$ has to decrease for ever.
La Salle's theorem now states that $V \rightarrow 0$ and hence the system is globally asymptotically stable.