The following second order system is given:
$\dot{x}_1 = -2x_1^3 + 2x_2^2x_1$
$\dot{x}_2 = x_1^2x_2 - 2x_2^3$
I want to determine whether the origin is globally asymptotically stable or not. For this, I need to construct a Lyapunov function. I tried $V(x) = 0.5x_1^2 + 0.5x_2^2$ but cannot prove that $\dot{V}(x) < 0$. What is the ideal Lyapunov function for this problem?
Let us consider the quadratic function you have suggested. Then, we have that the derivative of the Lyapunov function along the flow of the system is given by
$$ \dfrac{\partial V}{\partial x}f(x)=-2x_1^4+3x_1^2x_2^2-2x_2^4. $$ This can be rewritten as $$ \dfrac{\partial V}{\partial x}f(x)=\begin{bmatrix}x_1^2\\x_2^2\end{bmatrix}^T\underbrace{\begin{bmatrix}-2 & 3/2\\3/2 & -2\end{bmatrix}}_{\mbox{$M$}}\begin{bmatrix}x_1^2\\x_2^2\end{bmatrix}. $$
The fact is that the matrix is not positive definite, however this does not consider the fact that the vector $(x_1^2,x_2^2)$ is always nonnegative. We need to look at something called strict copositivity, which is luckily simple to establish in the 2-dimensional case.
A 2-by-2 matrix $A=[a_{ij}]$ is strictly copositive if and only if $a_{11},a_{22}>0$ and $a_{12}+\sqrt{a_{11}a_{22}}>0$. Applying this formula to the matrix $-M$ yields that the matrix $-M$ is indeed copositive and that
$$\dfrac{\partial V}{\partial x}f(x)<0\mathrm{\ for\ all\ }x\ne0$$
which proves the global asymptotic stability of the zero-equilibrium point of the system.
Edit. In fact, we can say more and without using the concept of copositivity. Start with
$$ \dfrac{\partial V}{\partial x}f(x)=-2x_1^4+3x_1^2x_2^2-2x_2^4. $$ and let $y_1=x_1^2$ and $y_2=x_2^2$, which yields the polynomial
$$ P(y_1,y_2):=-2y_1^2-2y_2^2+3y_1y_2. $$
Now pick the polar coordinates $y_1=r\cos(\theta)$ and $y_2=r\sin(\theta)$ where $\theta\in[0,\pi/2]$ and $r\ge0$. This yields
$$ P(y_1,y_2)=r^2(-2+3\cos(\theta)\sin(\theta))=r^2(-2+\frac{3}{2}\sin(2\theta))\le -r^2/2. $$
This implies that $$ \dfrac{\partial V}{\partial x}f(x)\le-\dfrac{1}{2}(x_1^4+x_2^4)\le-2V(x)^2. $$
This then implies that $$ V(x(t))\le \dfrac{V(x(0))}{1+2tV(x(0))} $$