Let $X : U \subset \mathbb R^n \to \mathbb R^n$ be a $C^1$ vector field and $x_0$ a singular point F $(i.e., X(x_0)=0)$. Let $h : V \to \mathbb R$ be a $C^1$ map defined on neighborhood $V \subset U$ of $x_0$ such that $h(x_0) = 0 $ and such that $\dot{h}(x) = \frac{d}{dt}h(\varphi(t,x))|_{t=0}>0$ for every $x \in V\setminus\{x_0\}$ (where by $\varphi(t,x)$ we denote the solution for $x'=X(x)$, $x(0) = x$). Suppose that for every neighborhood $W \subset V$ of $x_0$ there is $\tilde x \in W$ such that $h(\tilde x)>0$. Prove that $x_0$ is Lyapunov unstable.
I'll show what I've done so far: we suppose that $x_0$ is stable. We choose a compact neighborhood $V_0$ of $x_0$, since it is stable, then there exists an neighborhood $x_0\in V_1\subset V_0$ such that for any $x\in V_1$ we have $\varphi(t,x)\in V_0$ for every $t\geq 0$. By hypothesis we have $h(x)>0$ and since the positive semi-orbit of $x$ is such that $\varphi(t,x)\in V_0$ and $V_0$ is compact, then $\omega(x)\neq \emptyset$, compact and it is invariant (ie, $y\in \omega(x) \implies \varphi(t,y)\in \omega(x))$.
Now how to proceed? I've tried to use the fact that $\omega(x)$ is compact and hence $h$ would have a maximum at $\omega(x)$ and that didn't work. Any ideas, any help?
Following the structure of Arrowsmith & Place: "Dynamical systems ...", p92
Select some $R>r>0$ and some $x_1\in B(x_0,r)$ with $h(x_1)>0$. Then there is some neighborhood $U$ around $x_0$ where $h(x)<h(x_1)/2$ for all $x\in U$. As $h(φ(x_1,t))$ is an increasing function, this trajectory stays outside $U$.
Now $\bar B(x_0,R)\setminus U$ is a closed and thus compact set, the positive continuous function $\nabla h\cdot X$ has a positive minimum $K$ there. As long as the trajectory $φ(x_1,t)$ stays inside $\bar B(x_0,R)$, we conclude that $h(φ(x_1,t))\ge h(x_1)+Kt$. As $h$ is bounded on the compact set $\bar B(x_0,R)$, the trajectory has to leave this closed ball around $x_0$ of radius $R$.
Thus $x_0$ is unstable, as $r>0$ can be arbitrarily small and $R>r$ arbitrarily large.