I have the following differential system \begin{cases} x'(t) =y(t)+[1-x^2(t)-y^2(t)]x(t) \\ y'(t)=-x(t)+[1-2x^2(t)-y^2(t)]y(t) \\ \end{cases} How can I see if there exist some periodic solutions?
I only know Bendixson's theorem that gives me a sufficient condition for the non-existence of periodic solution..
On
If you want to find a circular trapping region, which works in many exercise examples, the first to consider is the radius dynamic $$ rr'=xx'+yy'=r^2(1-r^2)-x^2y^2. $$ Then use that $0\le x^2y^2\le \frac{(x^2+y^2)^2}4$ to find the inequalities $$ r\left(1-\frac54r^2\right)\le r'\le r^2(1-r^2) $$ so that $r'$ is guaranteed to be non-negative for $0\le r\le\frac2{\sqrt5}$ and non-positive for $r\ge 1$.
It remains to check that there are no stationary points of the system inside the annulus between these regions.
This is not exactly an answer to your question, as it does not constitute a proof. However, it is inconvenient to post it as a comment.
Running numerical simulations we can see that there is a single periodic solution, which is the $\omega$-limit of every other orbit (except for the constant orbit $x=y=0$).