Using Polar coordinates, show that the system \begin{align} \dot x &= x -y -x(x^2 + y^2) \\ \dot y &= x +y -y(4x^2 + y^2) \end{align}
gives $\dot r = r - r^3 (1+\frac34 \sin^2(2\theta))$.
You are given that this system has a single fixed point located at $(x, y) = (0, 0)$.
I understand that \begin{align} r\dot r &= x\dot x + y\dot y \\ &= x^2 -xy -x^2(x^2+y^2) +xy +y^2 -y^2(4x^2+y^2) \\ &= x^2 + y^2 -x^2(x^2 +y^2) -y^2(4x^2+y^2) \\ &= x^2 + y^2 -x^4 -y^4 -5x^2y^2. \end{align}
However, I cannot understand the final step to the answer.
It follows after this to find radii $r_1 < r_2$ such that $\dot r > 0$ for $r < r_1$ and $\dot r < 0$ for $r > r_2$. How does one go about this?
Hint.
Adding
$$ x\dot{x} = x^2 - x y -x^2(x^{2} + y^{2})\\ y\dot{y} = y x + y^2 -y^2(4x^{2} + y^{2}) $$
we obtain
$$ \frac 12\frac{d}{dt}(x^2+y^2) = x^2+y^2-x^2(x^{2} + y^{2})-y^2(4x^{2} + y^{2}) $$
substituting now
$$ x = r \cos\theta\\ y = r \sin\theta $$
we obtain
$$ \dot r = r-(1+\frac 34\sin^2(2\theta))r^3 $$
so we have an isolated unstable equilibrium point at $r=0$ and an stable limit cycle as can be depicted in the attached figure