Please, I need help or hint.
For the following system of ODEs:
\begin{equation} \dot{x} = -y+x \left(r^4-3r^2+1 \right) \\ \dot{y} = x+y \left(r^4-3r^2+1 \right) \end{equation}
where $r^2 = x^2 + y^2$. So, as a function of $x$ and $y$:
\begin{equation} \dot{x} = x^4 y + 2 x^2 y^3 - 3 x^2 y + x + y^5 - 3 y^3 + y \\ \dot{y} = x^5 + 2 x^3 y^2 - 3 x^3 + x y^4 - 3 x y^2 + x - y \end{equation}
How do I find stable / unstable limit cycle if I know that polynomial
\begin{equation} x^4-3x^2+1 \end{equation}
has 4 roots:
\begin{equation} x_1=\dfrac{1}{2} \pm \dfrac{\sqrt{5}}{2} \\ x_2= -\dfrac{1}{2} \pm \dfrac{\sqrt{5}}{2} \end{equation}
variable $\:r\:$ in the $\:\dot{x}, \dot{y}\:$ represents polar coordinates.
I have determined $\:\dot{r}=r^5-3r^3+r\:$ and $\:\dot{\varphi} = 1\:$ to analyze equilibria point [0,0] behavior, but I don't knowing that helps in this case.
$r$ as component of the polar coordinates is always positive. Thus you can ignore the negative roots. From $rr'=xx'+yy'$ you get the radius dynamic $$ r'=r(r^4−3r^2+1)=r(r^2+r-1)(r^2-r-1) $$ The sign of $r'$ is positive for small $r$, then negative after the first positive root $\frac{\sqrt5-1}2$, then positive again after the second positive root $\frac{\sqrt5+1}2$. In forward time, this means that $r$ is growing on the first segment, falling on the second, and growing again on the final segment, making the first positive root a stable equilibrium (for the radius) and the second an unstable one.