I am working on the following problem, given the system of two differential equations
$x′=2x+y−2x^3−3xy^2,$ $y′=−2x+4y−4y^3−2x^2y,$
So far, I have tackled similar problems by trying to find a trapping region that does not contain an equilibrium point in its interior. (Applying Poincare-Bendixson Theorem).
Clearly $(0,0)$ is an equilibrium point. (It will need to be shown it is the only equilibrium point). Now using a Lyapunov function, I am trying to find a trapping region.
You have a small mistake for your time derivative of the Lyapunov function, namely for $V(x,y)=2 x^2+y^2$ one gets $16 x^2 y^2$ instead of $20 x^2 y^2$, so
\begin{align} \dot{V}(x,y) &= 8(x^2+y^2)−8(x^4+2 x^2 y^2+y^4), \\ &= 8(x^2+y^2)-8(x^2+y^2)^2, \\ &= 8(x^2+y^2)(1-x^2-y^2). \end{align}
Now you want to find $\alpha$ and $\beta$ (with $0 < \alpha < \beta$) such that for the set $A=\{x,y \mid V(x,y) = \alpha\}$ it holds that $\dot{V}(x,y)>0$ and for the set $B=\{x,y \mid V(x,y) = \beta\}$ it holds that $\dot{V}(x,y)<0$. Namely, this would mean that all initial conditions on the level set $A$ of the Lyapunov function would flow towards higher level sets and that all initial conditions on the level set $B$ of the Lyapunov function would flow towards lower level sets. Thus all initial conditions starting on either $A$ or $B$ should end up in some region in-between $A$ and $B$.
It is worth noting that the level sets of $V(x,y)$ do not overlap with levels sets of $\dot{V}(x,y)$. So for $x^2+y^2=1$, where $\dot{V}(x,y)=0$, does not correspond with a level set of $V(x,y)$ and thus $x^2+y^2=1$ would not be a limit cycle.