The Chetaev theorem states
Consider the autonomous dynamical system $\dot{x}=f(x)$ and assume that $x=0$ is an equilibrium point. Let $V:D\rightarrow \mathbb{R}$ have the following properties:
(i) $V(0) = 0$
(ii) $\exists x_0 \in \mathbb{R}^n$, arbitrarily close to $x=0$, such that $V(x_0)>0$
(iii) $\dot{V}>0 \forall x \in U,$ where the set $U$ is defined as follows:
$U=\{x\in D:||x|| \leq \epsilon,$ and $V(x)>0\}$
Under these conditions, $x=0$ is unstable.
I would like to utilize this theorem to show that if $\alpha > 0$, the equilibrium point for the following system is unstable.
My solution is
It is clearly that if $\alpha > 0$, the $\dot{V}(\cdot)$ is positive definite but I don't know how to define the set $U$ to verify the condition (iii). Any clue?
Since for your case $\dot{V}$ is positive for all $x$ except the origin, you can take almost any $U$. For example, let us choose $\epsilon = 1$. Then the set $U$ is the whole disc $\|x\|\le 1$ except the origin (since at the origin $V(x)=0$). And at any point in the set $U$, you have $\dot{V}>0$.