Consider the system $$\frac{dx}{dt}=x (\lambda-x^2 + (1+\epsilon^2)y^2))+\omega y,~\frac{dy}{dt}=-\omega x + y (\lambda-x^2 + (1+\epsilon^2)y^2)).$$ Show that the system has a stable limit cycle for $\lambda, \epsilon >0.$
My approach: Setting the RHS of each equal to zero and adding the two terms yields: $$(x^2+y^2)\lambda - x^2 (x^2+y^2)-(1+\epsilon^2)(x^2+y^2)y^2=0.$$ Using polar coordinates $x=r \cos \theta,~y=r \sin \theta,$ we get $$r^2 \cdot (\lambda - r^2 \cos^2 \theta - (1+\epsilon^2) r^2 \sin^2 \theta)=0,$$ which gives $$r=0,~r=\pm \frac{ \sqrt{\lambda}}{\sqrt{(1+\epsilon^2 \sin^2 \theta)}}.$$ ............................................................................................
As one of the comments suggested, I tried setting the right hand sides of each equal to zero.
$$x (\lambda-x^2 + (1+\epsilon^2)y^2))+\omega y=0,~-\omega x + x (\lambda-x^2 + (1+\epsilon^2)y^2))=0.$$
I tried method of elimination to solve for $x$ and $y,$ but wasn't successful. I still don't know how can this be any simpler than what I did before. I'm still stuck in this problem. Can someone please explain me from this how can I proved the desired conclusion. Thank you for your time.
In polar coordinates the equations are $$ r'=r(\lambda-r^2\cos^2\theta-(1+\epsilon^2)r^2\sin^2\theta), \quad \theta'=-\omega. $$ For $\omega>0$ the second equation implies that the only equilibrium is the origin. Now, for $r$ very small $r'>0$, while for $r$ very large we have $$ r'=r(\lambda-r^2-\epsilon^2r^2\sin^2\theta)<0. $$ Hence, it follows from the Poincaré-Bendixson theorem that there is a periodic orbit. Its stability is also the desired one.