To study the system $$x' = −y +\epsilon x(x^2 + y^2)\\ y' = x +\epsilon y(x^2 + y^2),$$
the authors perform a change of coordinates, getting $$r'=\epsilon r^3\\ \theta'=-1.$$
So, they state that
Thus when $\epsilon>0$, all solutions spiral away from the origin, whereas when $\epsilon<0$, all solutions spiral toward the origin.
I cannot see this because, trying to solve the firt ODE, I found $$r^2(t) = \dfrac{1}{2(c-\epsilon t)}.$$
If $\epsilon>0$, the $r$ is not real for $t$ large.
Thank you in advance!
Your solution is correct,
$$r(t) = \frac{1}{\sqrt{r(0)^{-2} - 2\,\epsilon\,t}},$$
but the solution does grow unbounded when $\epsilon > 0.$ In this DE, your constant $c$ is necessarily positive as it is effectively a multiple of the initial distance from the origin. In fact, this DE has finite escape time for $\epsilon > 0$: the solution $r(t) \to \infty$ as $t\to \frac{1}{2\,\epsilon}\,r(0)^{-2}.$ When $\epsilon < 0,$ the solution $r(t) \to 0$ as $t\to\infty.$