We have the dynamical system $$ \begin{cases} x' =x(x^2+y^2-2x-3)-y \\ y' =y(x^2+y^2-2x-3)+x \end{cases} $$
the question is: is there a limit cycle for the above system? To solve this question, I have transformed it into polar coordinates I got $r'=r(r^2-2r\cosθ-3)$ Now, I do not know how to deal with the term '$\cosθ$'? Your help will be appreciated.
Making $x \dot x + y \dot y$ we have
$$ \frac 12\frac{d}{dt}(x^2+y^2) = (x^2+y^2)\left((x-x_0)^2+(y-y_0)^2-r^2\right) $$
with $x_0 = 1, y_0 = 0, r = 2$
this characterizes an attractive region for $(x-x_0)^2+(y-y_0)^2<r^2$ and an expansive region for $(x-x_0)^2+(y-y_0)^2>r^2$ with a limit cycle for $(x-x_0)^2+(y-y_0)^2=r^2$