How can the system $$\frac{dx}{dt}=-y+\epsilon x(x^2+y^2)$$$$\frac{dy}{ dt}=x+\epsilon y(x^2+y^2)$$ be transformed into $$\frac{dr}{dt}=\epsilon r^3$$ $$\frac{d\theta}{dt}=1$$ via polar coordinates?
I sub in $x=r\cos(\theta)$ and $y=r\sin(\theta)$ into both of the original equations, but there are $\frac{dr}{dt}$ and $\frac{d\theta}{dt}$ terms in both equations then. And adding/subtracting the equations doesn't seem to produce anything 'nice'.
Well, to obtain this form you can just differentiate $r^2(t) = x^2(t) + y^2(t)$ and $\theta(t) = \arctan{\frac{y(t)}{x(t)}}$. Let's check:
$$ \frac{dr^2}{dt} = 2 r \frac{dr}{dt} $$ but also $$ \frac{dr^2}{dt} = \frac{d}{dt}(x^2(t) + y^2(t)) = 2 x \dot{x} + 2 y \dot{y} = 2 \epsilon (x^2+y^2)^2 = 2 \epsilon r^4 $$ From this follows that $2 r \frac{dr}{dt} = 2 \epsilon r^4 $ and $\frac{dr}{dt} = \epsilon r^3$.
Similar with $\dot{\theta}(t)$:
$$ \frac{d}{dt} \left (\arctan{\frac{y}{x}} \right) = \frac{\dot{y}x - y \dot{x}}{x^2+y^2} \equiv 1 $$