For any $\alpha<0$, the critical point $(0,0)$ of the system $$\begin{cases} x'=\alpha x+y+x^2y\\ y'=-x+ay\cos(x) \end{cases}$$ is strictly stable.
Then I try two types of Liapunov function such as $ax^2+bxy+cy^2$, where $b^2-4ac<0$ and for another one is $x^{2m}+y^{2n}$. I found that the significant problem is tackling the positivity of $x^2y$.
Do you have any ideas
For the radius length you get $$ rr'=xx'+yy'=αr^2+r^4\cos^3\theta\sin\theta+αr\sin\theta(\cos(r\cos(\theta)-1) $$ Using $|\sin\theta|\le 1$ etc. and $0\le 1-\cos\phi=2\sin^2(\phi/2)\le \frac12\phi^2$, this gives an estimate of $$ rr'\le -|α|r^2+(1+|α|/2)r^4. $$ The right side is negative as long as $r^2\le\frac{|α|}{1+|α|/2}$.