We consider the nonlinear system $$x' = -x + y + xy $$ $$y' = x - y - x^2 - 2y^3 $$ and we want to investigate the nonlinear stability of the origin using Lyapunov functions. The hint is to consider a quadratic in terms of x and y.
So, I consider a function $V(x, y) = ax^2 + by^2 $ for $a,b\in\mathbb{R}$ to be determined. It is clear that $V(0,0) = 0$, regardless of our choice of $a,b$. So, it remains to show the time derivative is non-positive. After some simplification, we have that $$\frac{dV}{dt} = \nabla V \cdot (x', y') = -2ax^2 -2by^2 + (2a + 2b)xy + (2a - 2b)x^2y - 4by^4 $$ If we can find $a, b > 0$ such that the terms $xy$ and $x^2y$ zero out, we've succeeded. This would require that $$2a + 2b = 0 $$ $$2a - 2b = 0 $$ But this implies that $a = b = 0$, and so then our Lyapunov function isn't positive definite, so this is a contradiction. Does this mean the Lyapunov function doesn't exist? How can I proceed forward?
When evaluating whether a polynomial functions is positive or negative definite it can be useful to group terms of the same order and evaluate each group separately. Here with order I mean expression of the form $\prod_i x_i^{n_i}$ for $x_i$ each state of the system, $n_i\in\mathbb{Z}$ and the order $k=\sum_i n_i$. So for your system with two states this means terms of the form $x^p\,y^q$, with $k= p+q$. Usually it is preferred to not have terms of odd order ($k$ being odd). Though, this is not a hard requirement, for example consider $-x^2+x^3-x^4$ which is negative definite in $x$.
However, in your case when only setting the third order term in $V'$ to zero gives only the constraint
$$ 2\,a - 2\,b = 0. $$
Substituting in $b=a$ and simplifying the second order terms yields
\begin{align} \frac{dV}{dt} &= -2\,a \left[x^2 + y^2 - 2\,x\,y + 2\,y^4\right], \\ &= -2\,a \left[(x - y)^2 + 2\,y^4\right]. \end{align}
The second and fourth order groups individually are only negative semi-definite. However, it can be shown that their combination is negative definite in $x$ and $y$.