The problem is that after a seemingly trivial change of coordinates I get spurious solutions and I can't explain why.
The ODE is \begin{equation} \ddot x+a\dot x(x^2+\dot x^2-1)+x=0 \end{equation} which is equivalent to the first-order system \begin{cases} \dot x=y \\ \dot y=-x-ay(x^2+y^2-1) \end{cases}
By taking the polar coordinates, I get the equations \begin{cases} \dot r\cos\theta-r\dot{\theta}\sin\theta=r\sin\theta \\ \dot r\sin\theta+r\dot{\theta}\cos\theta=-r\cos\theta-ar\sin\theta(r^2-1) \end{cases}
Then, by equating the coefficients of the sines and the cosines, I get at once both these equations for $\dot r$
\begin{equation} \dot r=0 \end{equation} \begin{equation} \dot r=-ar(r^2-1) \end{equation}
I have a feeling that I should discard $\dot r=0$, but I'm not able to provide an explanation for this. Why should I discard that equation and the solutions it brings?
Rather than comparing coefficients that are not constant, consider this as a linear system in $\dot r$, $r\dot θ$, \begin{align} \pmatrix{\cosθ&-\sinθ\\\sinθ&\cosθ}\pmatrix{\dot r\\r\dot θ} =\pmatrix{r\sinθ\\-r\cosθ+ar(r^2-1)\sinθ} \end{align} As the matrix is a rotation, you can solve this by multiplying with the transpose matrix.
You could of course also use that $$ r\dot r=y\dot y+x\dot x=\dot x(\ddot x+x)=-a\dot x^2(r^2-1) \\\implies \dot r=a(1-r^2)r\sin^2θ $$ and $$ r^2\dot θ=x\dot y-y\dot x=-r^2+x(\ddot x+x)=-r^2-ax\dot x(r^2-1) \\\implies \dot θ=-1+a(1-r^2)\sinθ\cosθ $$