Given
$$\left\{\begin{aligned} x' &= -x^3 + 7xy^2\\ y' &= -3x^2y+y^3\end{aligned}\right.$$
find $a, b > 0$ such that $L(x,y) = a x^2 + b y^2$ obeys $\frac{d}{dt}L \neq 0$ whenever $(x,y) \neq (0,0)$. State whether the origin is a stable or unstable equilibrium.
So we want to find $a,b$ where $L(x,y)$ is a Lyapunov function, and then determine the stability of the equilibria.
First differentiate: $\frac{dL}{dt}=2axx'+2byy'$
Now plug in system: $2ax(-x^3+7xy^2)+2by(-3x^2y+y^3)$
By checking the equilibria, we see that both $x'$ and $y'$ equal zero only when $(x,y)=(0,0)$.
Therefore we need to find when $2ax(-x^3+7xy^2)= -2by(-3x^2y+y^3)$
This is where I am confused, as I am not sure how to find $a$ or $b$ purely in terms of $x,y$ and what restrictions I may need to impose. Any help is appreciated. Thank you.
Complete the square in $$ \dot L=2(-ax^4+(7a-3b)x^2y^2+by^4)=\frac1{2a}\left(-(2ax^2-(7a-3b)y^2)^2+(4ab+(7a-3b)^2)y^4\right). $$ This means that there will always be points in any neighborhood of $(0,0)$ with positive and negative signs of this type of Lyapunov function.
Checking with Wolfram alpha
confirms the saddle point behavior in the plot