$\lambda \in \mathbb{R}$. Find $a,b$ such that $V(x,y)=ax^2+by^2$ is Lyapunov with $$\begin{cases} x ' = \lambda x\\ y'= x+\lambda y \end{cases} $$ and find the stability of $(0,0).$
$V'=2ax(\lambda x)+2by(x+\lambda y)=2\lambda a x^2+2byx+2b\lambda y^2$. This quadratic form has the matrix $ \left( \begin{array}{ccc} 2\lambda a & b \\ b & 2\lambda b \\ \end{array} \right)$ the characteristic polynomial is $(2\lambda a-z)(2\lambda b -z)-b^2 = 4\lambda^2 ab - z(2\lambda a+2\lambda b)+z^2 -b^2= z^2-z(2\lambda a + 2 \lambda b) + (4\lambda ^2 ab-b^2).$
The roots are $z=\frac{2\lambda a + 2\lambda b \pm \sqrt{4\lambda ^2 a^2 + 8 \lambda ^2 ab + 4 \lambda ^2 b^2 - (16 \lambda ^2 ab-8\lambda^2 a b^3+b^4)}}{2}$
But this seems hard to reduce in order to find the roots.
How can I find the stability of $(0,0)?$ I was thinking of finding the zeroes and then determine stability according to the sign of $V$.
Set $b=4|λ|$ to get $$ \frac12\dot Vf=λax^2 + sign(λ)·((x+2λy)^2- x^2)\\ =sign(λ)·((|λ|a-1)x^2+(x+2λy)^2) $$ so you need only chose $a>\frac1{|λ|}$ to get a definite result.