Cetaev Theorem states that:
"Considering the ode $X'=F(X)$, with an equilibrium $x_{0}$.
If there is a function $V$: $U_{0} \rightarrow $ IR and a region $\omega$ in $U_{0}$ that contains $x_{0}$, such that:
(1) $V$ is positively defined;
(2) $V'$ is also positively defined;
(3) $V=0$ in the border of $\omega$.
Then $x_{0}$ is an unstable point."
My question is, for example, if I define $V(x,y)=x^{2}+y^{2}$, V and V' were positively defined on $\omega=\mathbb{R}^{2}$, then this has no boundary but satisfies (1) and (2), so does it still work to conclude that $x_{0}$ is an unstable point?
Thanks in advance.
What you have given is not a correct formulation of Chetaev theorem. However, the question is clear and the answer is yes. The statement in question can be proven in a manner similar to the proofs of Lyapunov's theorems.
Consider any non-zero solution $(x(t),y(t))$, $x(t_0)=x_0$, $y(t_0)=y_0$. Since for any non-zero point $(x,y)$ we have $\dot V(x,y)>0$, the function $V(x(t),y(t))$ monotonically increases along any trajectory (except for the zero solution). This means that the solution does not tend to zero. But then there exists the number $$ m= \inf\limits_{t\ge t_0}\, \dot V(x(t),y(t)),\quad m>0 $$ ($\dot V$ is positive definite, so $\dot V=0$ only at $(x,y)=0$).
For any solution $$ \forall t\ge t_0\quad V(x(t),y(t))=V(x_0,y_0)+\int_{t_0}^t \dot V(x(t),y(t))\, dt\ge V(x_0,y_0)+m(t-t_0). $$ Hence, $V(x(t),y(t))$ grows unlimited; this means that the solution eventually leaves any predetermined neighborhood of the origin, therefore, the system is unstable.