Given $V(x,y)=ax^2+by^2$ I'm asked to find $a$ and $b$ to make $V$ a Lyapunov function for the following systems:
$(1)$\begin{cases} x'= -x^\color{red}{3}+xy^2 \\ y'= -\color{red}{2}x^2y-y^3\end{cases}
Here I have $\dot{V}(x,y)=2ax(-x^2+xy^2)+2b(-y^3)=2ax^3+2ax^2y-4bx^2y^2-2by^4$.
The first two termins can take positive and negative values depending on $x$, then here I would set $a=0$; then $b>0$ to make it $\dot{V}(x,y)\leq 0$.
$(2)$\begin{cases} x'= -x^\color{red}{3}/2+2xy^2 \\ y'= -y^3\end{cases}
In this case is $\dot{V}(x,y)=2ax(-x^2/2+2xy^2)+2b(-y³)=-ax^3-x^2y^2-2by^3$.
For the middle term is $-x^2y^2\leq 0$, but $-ax^3-2by^3$ could be less or greater than zero for any $a,b$ chossing an appropiate $x,y$. Then in this case $a=b=0$ is the only possible solution?
UPDATE: I copied the problem from my notes instead of the original problem, doing this I messed up some of the exponents. The fixed exponents are those in red, which I before miscopied as 2; I also missed a constant.
Solution for Updated Problem
The phase portrait for the first problem is given by (single critical point at origin):
The phase portrait for the second problem is given by (single critical point at origin):
For:
$$V(x,y)=ax^2+by^2$$
The derivative is:
$$V'(x, y) = 2 a x x' + 2 b y y'$$
Now substitute in $x'$ and $y'$, simplify and choose $a$ and $b$ accordingly.
For (1), we have:
$$V'(x, y) = -2 a x^4 +x^2 y^2(2a-4b) -2by^4$$
We need to choose $a$ and $b$ such that:
$$V(x, y) > 0~\mbox{and}~V'(x, y) \lt 0 ~ \forall ~(x, y) \ne (0,0)$$
Do you see how to choose $a$, $b$ to satisfy this? For example, $a > 0, b > 0$ with $a = 2b$. That first choice eliminates the middle term. You could also choose $a > 0, b > 0, a \lt 2b$ (all terms negative).
For $(2)$, we have:
$$V'(x, y) = -a x^4 + 4 a x^2 y^2 - 2 b y^4$$
What do you need to show the second one? Recall, you can not choose $a = 0$ or $b = 0$, else you do not have a valid Lyapunov Function.