I have converted the system of ODEs, $$x'=x-y-x(x^2+5y^2)$$ $$y'=x+y-y(x^2+y^2),$$
to polar coordinates and got this:
$$ r' = r-r^3(1+4\sin^2(\theta)\cos^2(\theta))$$ $$\theta'=1+4\cos(\theta)\sin^3(\theta).$$
Now I need to find two circles, $r_1$ and $r_2$, such that $0<r_1<r_2$, where $r'>0$ on $r_1$ and $r'<0$ on $r_2$. I have no idea how to do that in this case, as I just don't see how to "exctract" any circles from this system. Some help would be appreciated.
Since $0\leq\sin^2\theta\cos^2\theta\leq 1$. Thus, $1\leq 1+4\sin^2\theta\cos^2\theta\leq 5$. Thus, if you can find an $r$ sufficiently small that $r>5r^3$, you can let this be $r_1$. And any $r$ with $r<r^3$ will serve as $r_2$. Note that $r(t)=r_q$ and $r(t)=r_2$ are not solutions to this ODE, just radii which help you establish a forward-invariant anulus, and a periodic orbit therein.