The Lagrange-Dirichlet Theorem is partially reversed by the following result:
- if the costraints are holonomous, bilateral, ideal,
- if $q^*$ is a critical point for the potential energy $U$,
then the equilibrium $q^*$ is Lyapunov stable IFF the (quadratic form associated to the) Hessian matrix of $U$ is positive definite.
It seems intuitive, but I don't know how to prove it. Should I use first and/or second Lyapunov method?
First of all let's me say that a stable equilibrium position is every isolated local minimum of $U$, even if the minimum cannot be deducted from the Hessian of $U$, begin this null (consider the 2D example $U = q_1^4+q_2^4$.)
In the case the Hessian of $U$ is not null and is not positive definite, this means that at least one of its eigenvalues is negative, then in the study of small oscillations around $q^*$, the fundamental motion related to this eigenvalues is not a sinusoidal oscillation, but is an exponential solution in which the system depart from $q^*$. This is in some way related to the first Lyapunov method.