I got the system: $$\dot{x} = x-y-y^3-2x(x^2+y^2)$$ $$\dot{y}= x+y-2y(x^2+y^2)$$ And it's given that the origin is the only fixed point. I've converted it to Polar using $\dot{r}r=\dot{x}x+\dot{y}y$ and ended up with: $$r\dot{r}=r^2-2r^3-r^4\cos\theta \sin^3\theta $$ Which I have simplified to the following but I do not trust my trig within this. $$\dot{r}=r-2r^2-\frac{r^3}{4}(\sin2\theta-\frac{1}{2}\sin4\theta)$$ I'm not very sure if this was the right way to go because when it comes to classifying $\dot{r}$ now I have to deal with $\sin2\theta$ and $\sin4\theta$. So I guess my question right now is did I end up with the correct $\dot{r}$? If yes how do I end up trapping the region in this situation?
I know that I have to check for when $\dot{r}>0$ for the outward flow and $\dot{r}<0$ for the inward flow. But I do not know how to inspect that with both $\sin{2\theta}$ and $\sin{4\theta}$ present at the same time.
My brain right now is a scattered mess and I do not trust anything that comes out of it so I have resorted to share this with you and hopefully someone can put my mind to rest and let me know where I am going wrong because I have tried to solve this question 4 times now and every time I get a different $\dot{r}$.
You missed a power in combining the last terms of the equations, you should have got $$ r\dot r = r^2-2r^4-xy^3=r^2-(2+\sin^3θ\cosθ)r^4 $$ so that because of the mean inequality of the geometric and quadratic mean $$\sqrt[4]{|\sqrt3cs^3|}\le \sqrt{\frac{3c^2+s^2+s^2+s^2}4}=\sqrt{\frac34}$$ we get bounds for the derivative of the radius in terms of the radius only as $$ r-ar^3\le \dot r\le r-br^3 ~~\text{ with }~~ a,b=2\pm \frac{3\sqrt3}{16}. $$ This means that the radius of a solution decreases in time where the right side is negative, which is for $r>\sqrt{1/b}\approx 0.773$. The radius increases in time where the leftmost term is positive, that is for $0<r<\sqrt{1/a}\approx 0.656$.
This gives an invariant annulus or trapping region. Per the given claim of no other stationary points outside the origin one concludes the existence of a limit cycle.
The plot shows clearly the attracting limit cycle in the trapping region $0.6<r<0.8$, visually it is inside the smaller annulus $0.7\lessapprox r\lessapprox 0.75$.