I have $$\frac {\mathrm{d}x}{\mathrm{d}t}=x-y-(2x^2+y^2)x $$$$\frac{\mathrm{d}y}{\mathrm{d}t}=x+y-(x^2+2y^2)y$$
I have calculated $\frac{\mathrm{d}r}{\mathrm{d}\theta} =r+r^3(\frac12\sin^2(2\theta)-2) $
I would appreciate if someone could check this and see if they get the same answer
When I swap the ODE system into polars I get
\begin{align} \frac{dr}{dt} & = r - 2r^3 \left[ 1 + \frac{1}{4} \cos(4 \theta) \right] \\ r \frac{d \theta}{dt} & = r + \frac{r^3}{4} \sin(4 \theta) \\ \end{align}
And I can't easily see how this will collapse (through use of the chain rule) to give your result.