I am reading the book of Hassan KhalinlNonlinear Systems (chapter 8.1. Center Manifold Theorem).
In Example 8.2 the author states a system
$$\dot{y}=yz$$ $$\dot{z}=-z+ay^2,$$
in which $a\in \mathbb{R}$.
As the linearized system at $(0,0)$ is already diagonal and y is associated with the eigenvalue which is zero. We can use $z=h(y)$, plugging this into the second equation we obtain the center manifold equation
$$\dfrac{\partial h(y)}{\partial y}\dot{y}+h(y)-ay^2=0.$$
Using the first equation and $z=h(y)$ we obtain:
$$\dfrac{\partial h(y)}{\partial y}yh(y)+h(y)-ay^2=0.$$
Now, he assumes that $h(y)=O(|y|^2)$ using this in the first equation we obtain
$$\dot{y}=yO(|y|^2)=O(|y|^3)$$
from which it is not possible to draw any conclusions.
Now, he assumes that $h(y)=h_2y^2+O(|y|^3)$. We have to determine the coefficient $h_2$ by using the center manifold equation which yields $h_2=a$. Using this for the first equation we obtain
$$\dot{y}=ay^3+O(|y|^4).$$
The author now states that $a<0$ leads to an asymptotically stable origin and $a>0$ to an unstable equilibrium point at the origin.
Question 1: From a $y-\dot{y}$-plot it is evident that the origin is asymptotically stable. Is there a more rigorous way to show this? I tried to construct a Lyapunov function $$V(y)=0.5y^2 \implies \dot{V}=y\dot{y}=ay^4+O(|y|^5).$$
I know that for the case of multivariable functions $V$ the higher order terms do matter for the assessment of positiveness/negativeness. But I don't see a reason why this should not be true for the single variable case. Hence, I would think that using this Lyapunov function would be a rigorous way to show the asymptotic stability.
Then the author also investigates the case $a=0$. It is stated that this implies that $h(y)=0$ and thus $\dot{y}=0$ which implies a stable origin for the nonlinear system.
If I use the center manifold equation ($a=0$) I obtain
$$\dfrac{\partial h(y)}{\partial y}yh(y)+h(y)=0 \implies \left[h'(y)y+1 \right]h(y)=0.$$
Question 2: The solution to this equation is given by $h(y)=0$ and $h(y)=\ln\dfrac{C}{y}$. Don't we consider the second equation because it does not fulfil $h(0)=h'(0)=0$ and is not defined at $y=0$?
For your first question, you don't need any theorem, in the sense that you have already all you need. More precisely, say that $a>0$ (the case $a<0$ is analogous). For $y>0$ sufficiently small we have $$ \dot y=ay^3+O(y^4)\ge ay^3−cy^4=y^3(a−cy)>0, $$ while for $y<0$ sufficiently small we have $$ \dot y=ay^3+O(y^4)≤ay^3+cy^4=y^3(a+cy)<0. $$ Note that this gives the sign of $\dot y$ and so the origin is unstable.
For your second question, as I have already mentioned you are completely right. Any center manifold is tangent to the center space and so we must have $h(0)=h′(0)=0$.