Given a (autonomous) dynamical system, one can prove instability of a point via the Lyapunov method, by simply finding a Lyapunov function that increases in a neighbourhood of the point. This ensures a stronger property than instability. It ensures the existence of a small enough neighbourhood, such that all orbits starting inside exit the neighbourhood at some time (instability only requires the existence of one such orbit, quite weaker).
My question is the following: does this stronger property allow that the system might converge (or approach) to the point in question? In other words, can a dynamical system converge (or approach) to a point admitting a neighbourhood with (strictly) increasing Lyapunov function? Thanks for any advice or reference.
EDIT: So far we proved that this cannot happen for continuous time systems. Is this also true for discrete time systems? Since the answer is in general settled to a no (continuous systems offer a counterexample as per the answer below) I will raise as a separate question the discrete case : Chetaev theorem for discrete time.
The answer to this question depends on the flavour of theorem you wish to invoke. For instance, here is a theorem from a nonlinear control text one could use,
Indeed, this flavour only demands that there exists a one-dimensional unstable manifold passing through the origin (condition 2) since the positive definiteness is only assured on some set of points arbitrarily close to the origin. This does not preclude stability towards that set, for instance, including the origin along some set of measure zero. However, if you ask $V$ be locally positive definite except at $x = 0$, then that variant does not support convergence to the origin since all the points near the origin are expelled. The text I follow makes this comment:
So it boils down to what theorem you are choosing to use. In [2] there is further detailed comment to this effect for a very similarly stated theorem (although, not as precise as Sastry or Khalil were, so I will not quote the theorem).
The reason for this of course derives from the proof of instability. In proving Theorem 5.29, within any ball one picks $x(0)$ so that $V(0) > 0$. If it is the case that $V(0) > 0$ in an open neighbourhood of $0$ excluding $0$, then the proof can be applied to show all trajectories exit the neighbourhood.
[1]: S. Sastry. Nonlinear Systems: Analysis, Stability and Control. 1999.
[2]: W. M. Haddad and V. S. Chellaboina. Nonlinear Dynamical Systems and Control. 2008.