Since there is no systematic method to find Lyapunov functions, how should I approach the question shown below to find corresponding Lyapunov function? With $\alpha<0$
$$\begin{align} x_1'&=-3x_2\\ x_2'&=x_1-\alpha(2x_2^3-x_2)\\ \end{align} $$
Since there is no systematic method to find Lyapunov functions, how should I approach the question shown below to find corresponding Lyapunov function? With $\alpha<0$
$$\begin{align} x_1'&=-3x_2\\ x_2'&=x_1-\alpha(2x_2^3-x_2)\\ \end{align} $$
Copyright © 2021 JogjaFile Inc.
As @CTNT proposed first linearize the system at the origin to obtain the system matrix $\boldsymbol{A}$ of the linearized system
$$\boldsymbol{A}=\begin{bmatrix}0 & -3\\1 & \alpha \end{bmatrix}.$$
The linearized system is asymptotically stable (also called Hurwitz) as the characteristic polynomial is equal to
$$\chi(s)=s^2-\alpha s +3.$$
By the Hurwitz criterion for second order polynomials (all coefficients have to be positive; note, that $-\alpha>0$), we know that the linearized system is asymptotically stable. By the indirect method of Lyapunov (as we have a polynomial right-hand side of the differential equation system all conditions for linearization are fulfilled in a neighbourhood of the origin) we know that the origin of the nonlinear system is also asymptotically stable.
By the Lyapunov's converse theorems we also know that there exists a unique symmetric positive definite matrix $\boldsymbol{P}$ as a solution to the Lyapunov equation
$$\boldsymbol{PA}+\boldsymbol{A^TP}=-\boldsymbol{Q},$$
in which $\boldsymbol{Q}$ is a positive definite matrix. As @CTNT proposed $\boldsymbol{Q}=\boldsymbol{I}$ is often chosen. If you solve the system you will have found a Lyapunov function for asymptotic stability for the origin.
An alternative approach uses the standard Lyapunov candidate function
$$V=\dfrac{1}{2}\left[x_1^2+kx_2^2\right]$$ $$ \implies \dot{V}=x_1\dot{x_1}+kx_2\dot{x}_2=-3x_1x_2+kx_2\left[ x_1-\alpha\left(2x_2^3-x_2 \right)\right]$$ $$=-3x_1x_2+kx_1x_2-\alpha k\left(2x_2^4-x_2^2 \right)$$
Now, set $k=3$ to eliminate the cross-product term to obtain
$$\dot{V}=-\alpha k(2x_2^4-x_2^2)=\alpha k x_2^2(1-2x_2^2).$$
For $1-2x_2^2>0 \implies |x_2|<\dfrac{\sqrt{2}}{2}$ we know that
$$\dot{V}\leq 0.$$
By using Barbashin and Krasovskii Theorem in the neighbourhood $D=\{x\in \mathbb{R}^2|x_1^2+x_2^2<1/2\}$ of the origin we see that no solution can stay inside the set $S=\{x\in D\,|\,\dot{V}=0\}=\{x\in D\,|\,x_2=0\}$, other than the trivial solution $x_1=0$ and $x_2=0$ (verify by plugging this into the differential equation). This implies that the origin is asymptotically stable (which we already knew by the indirect method of Lyapunov). This method does not require to solve the Lyapunov equation but at the same time has the downside that a more advanced stability theorem needs to be applied.
Looking at the phase portait (for a specific $\alpha=-10$) suggests that the system has an unstable limit cycle which makes it very unlikely that the origin is globally aymptotically stable for arbitrary $\alpha$. Note, that this is not a rigoruous statement it is just an observation. If you can prove that there exists a limit cycle then you can make this statement rigoruous.