This was a problem that I encountered about a year ago: Use the Lyapunov function method to determine the stability of the equilibrium of the origin of this system:
$$(x_1)' = -x_1 + x_2 - x_{1}^{3}$$ $$(x_2)' = x_1 - 2x_2 + x_{2}^{2}$$.
I tried some candidate functions: $V(x, y) = ax^2 + by^{2}$, $\nabla V = (2ax, 2by)$ so $\dot{V} = (2ax, 2by)\cdot(-x+y-x^{3}, x-2y+y^{2}) = 2ax(-x + y - x^{3}) + 2by(x - 2y + y^{2}) = -2ax^{2} + 2axy - 2ax^{4} + 2bxy - 4by^{2} + 2by^{3} \leq 0$ and so $(2a + 2b)xy + (2b)y^{3} \leq 2ax^{2} + 2ax^{4} + 4by^{2}$ for $(x,y)$ in some open $U$ containing $(0, 0)$, but if $2a + 2b = 0$, $a = -b$ and so $V(x,y) = ax^{2} - ay^{2}$ is not positive definite in any open $U$ around $(0, 0)$. Further if $b = -a$, $-2ax^{2} - 2ax^{4} + 4ay^{2} - 2ay^{3}$ is not negative definite in any open $U$ around $0$.
Some others I've tried: $$V(x,y) = ax + by + cx^{2} + dy^{2} + ex^{4} + fy^{4}$$ $$V(x,y) = ax^{2} + by^{2}e^{2x}$$
The function $V(x,y)=\frac{x^2+3y^2}{2}$ gets the job done, as
$$V_x=x, V_y=3y$$
$$\begin{align} \frac{dV}{dt} &=x(-x+y-x^3)+3y(x-2y+y^2) \\ &=-x^2+4xy-6y^2+3y^3-x^4 \\ &= -(x-2y)^2+(-2y^2+3y^3)-x^4 \\ & = -(x-2y)^2-2y^2(1-\frac{3}{2}y) - x^4 \end{align}$$
Thus whenever $|y|< \frac{2}{3}$, we have $1-\frac{3}{2}y\ge 1- \frac{3}{2}|y|> 0$, hence $\frac{dV}{dt}<0$ unless $x=y=0$. So the zero solution is asymptotically stable.
To come up with this, just let $V(x,y)=\frac{x^2+ay^2}{2}$ (you don't need to have $V(x,y)=ax^2+by^2$, as we can always factor $a$ out), we get $$\frac{dV}{dt} = -x^2+(a+1)xy-2ay^2+ay^3-x^4$$
Just ignore $-x^4$, and try to make sure $-x^2+(a+1)xy-2ay^2$ is negative definite, and after completion of squares, there is some negative $y^2$ left to dominate $y^3$.
But of course, it's straightforward to recognize the linearized system $$\begin{pmatrix} x \\ y \end{pmatrix} ' =\begin{pmatrix} -1 & 1 \\ 1 & -2 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix}$$
is asymptotically stable around $0$ as both of the eigenvalues of the matrix are real negative, while $x^3, y^2$ are negligible compared to the linear term.