We take a look at the system $$ x'=y^3, \quad y'= \cos(x)\sin(x). $$ It has a Hamiltonian $H(x, y) = \frac{1}{4}y^4 - \frac{\sin(x)^2}{2}$ for $(x, y) \in \mathbb{R}^2$.
It is clear that the points $n\pi$ ($n \in \mathbb{Z}$) are Lyapunov-stable equilibrium points (use Lyapunov function $H$). The equilibrium points $\frac{n \pi}{2}$ ($n \in \mathbb{Z}$) however are not so easy to handle. It is intuitively clear to me, that these are unstable, since $H$ suggests that the orbits close to these equilibra are approximately ellipses centered around the Lyapunov-stable equilibria. So I suggest that the familiy $\frac{n \pi}{2}$ consists of unstable equilibria.
Is there any mathematically rigorous way to show this, maybe using the definition of stability? Linearization and a direct Lyapunov method using $H$ don't help here as far as I'm concerned.