We have the following system: \begin{cases} x'=x(x^2+y^2-2x-3)-y\\ y'=y(x^2+y^2-2x-3)+x \end{cases} We want to determine that it has a single limit cycle. To prove this we had to use the Poincare Bendixson theorem.
We have the linearized system: \begin{cases} x'=-y\\ y'=x \end{cases} $A=\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$. We can see that we have a center point.
After that I found the stationary points. My problem arose when I had to determine the set A, which does not contain stationary points and is invariant to flow. I tried to solve this problem, but without any success.

As was established, for the dynamic of the radius you get $$ \dot r=r(r^2-2x-3). $$ This can be weakened to a scalar differential inequality $$ r((r-1)^2-4)\le\dot r\le r((r+1)^2-4)\\ r(r-3)(r+1)\le r\le r(r-1)(r+3). $$ The lower bound is positive for $r>3$, the upper bound negative for $r<1$. This means that the annulus $1\le r\le 3$ is a trapping region in negative time direction. As the angular velocity around the origin is always $1$, there are no equilibrium points except the origin. This establishes the existence of a repelling limit cycle inside the annulus. However not that it is single, there could in theory also be a triple of repelling, attracting and again repelling limit cycles.
Numerical experiments suggest that the limit cycle is close to the circle of radius $2$ around $(1,0)$, that is the root locus of the factor $x^2+y^2-2x-3$. Let $R=\sqrt{(x-1)^2+y^2}$, then $$\begin{align} \dot RR&=(x-1)\dot x+y\dot y=((x-1)x+y^2)(R^2-4)-(x-1)y+yx\\ &=R^2(R^2-4)+(x-1)(R^2-4)+y \end{align}$$ By Cauchy-Schwarz $$ |(x-1)(R^2-4)+y|\le R\sqrt{(R^2-4)^2+1} $$ so that $$ R(R^2-4)-\sqrt{(R^2-4)^2+1}\le \dot R\le \underbrace{R(R^2-4)}_{=f(R)}+\underbrace{\sqrt{(R^2-4)^2+1}}_{=g(R)} $$
The upper bound is negative on $[1.1,1.8]$, the lower bound positive for $R>2.2$, so that the annulus $1.8\le R\le 2.2$ is a trapping region in negative time direction. The plot of the numerical solution shows that this bound is rather tight.