$$\dot x=y+x[\mu-(x^2+y^2-1)^2]$$
$$\dot y=-x+y[\mu-(x^2+y^2-1)^2]$$
I used the polar coordinates substitution :
$$x=r\cos\theta$$ $$y=r\sin\theta$$
and via the expressions :
$$r\dot r=x\dot x+y\dot y$$
$$\dot\theta=\frac{x\dot y−y\dot x}{r^2}$$
Using a polar coordinate substitution correctly for $x$ and $y$, one yields : :
$$\dot r=r(\mu-(r^2-1)^2)$$
$$\dot\theta=-1 $$
I found: $$r=0, r=\pm\sqrt{1+\sqrt\mu} ,r=\pm\sqrt{1-\sqrt\mu} $$
Which one of these limit cycles is stable and which one is unstable and why?
The first factor $r$ is always positive for $r>0$.
The second factor has value $μ−1$ at $r=0$. At each positive root it will switch sign, until after the upper root the term becomes negative.
This gives a stable cycle at the upper root. The lower root is either not real or results in an unstable cycle.