I am still new to Lyapunov stability and I have a question:
The system is:
$\dot{x}_1 = x_2(1-x_1^2)$ and $\dot{x}_2=-(x_1+x_2)(1-x_1^2)$
I used $V(x)=\frac 1 2(x_1^2+x_2^2)$ Then, I get $\dot{V}(x)= -(1-x_1^2)x_2^2$.
Can I define $X=\{x\in \mathbb {R}^2 | x_1^2+x_2^2<1\}$ where $\dot{V}(x)<0$ $\forall x\in X$ and conclude it is asymptotically stable at the origin?
What I am curious is how I define the open set X in which $\dot{V}(X)<0$. Can it be arbitrary as long as it is open and contains $0$?
Your function $V$ is a weak Lyapunov function in the region $\Omega = \{ |x_1|<1 \}$, since $V$ is positive definite and $\dot V \le 0$ in $\Omega$, but it's not a strong Lyapunov function in $\Omega$, since $\dot V = 0$ along the whole line $x_2=0$ (so that the condition “$\dot V$ is negative definite in $\Omega$” is violated). So you can't use Lyapunov's theorem to show asymptotic stability.
But asymptotic stability follows instead from LaSalle's theorem, since the required extra assumption for that theorem is satisfied. Namely, on the line segment $(x_1,0)$ with $|x_1|<1$ (the “problematic” part of $\Omega$, where $\dot V$ vanishes), the ODEs reduce to $$(\dot x_1, \dot x_2) = (0, -x_1(1-x_1^2)),$$ which, because of the nonzero second component, makes it impossible for any solution except the equilibrium solution $(x(t),y(t))=(0,0)$ to stay in that line segment for all $t$.
But a simpler alternative is to note that in the region $\Omega$, where the factor $1-x_1^2$ is positive, the trajectories of your system agree up to time reparametrization with the trajectories of the linear system $\dot x_1 = x_2$, $\dot x_2 = -(x_1+x_2)$. And determining stability of that system is something that I think you know how to do!