I have the system
$x^\prime = x-y-x(x^{2}+y^{2})$
$y^\prime = x+y-y(4x^{2}+y^{2})$.
I want to show with polar coordinates that I eventually end up with
$r^\prime = r-r^{3}(1+ \frac{3}{4}$sin$^{2}(2 \theta))$.
I have used the following substitutions: $x = r$cos$(\theta)$, $y=r$sin$(\theta)$, $x^{2} + y^{2} = r^{2}$ and $rr^\prime = xx^\prime + yy^\prime $.
I have made a few attempts but my recent one is as follows: \begin{align} rr^\prime &= xx^\prime + yy^\prime \\ &=x(x-y-x^{3} -xy^{2}) + y(x+y-4x^{2}y^{2}+y^{3}) \\ &=x^{2} -xy + x^{4} + x^{2}y^{2} + yx + y^{2} - 4x^{2}y^{2} + y^{4} \\ &=r^{2} + (x^{4} + y^{4}) - 3x^{2}y^{2} . \end{align}
I am not sure if I am heading in the right direction. I also have noted that $\sin(2\theta) = 2\sin(\theta)\cos(\theta) $.
Any help would be much appreciated.
We are given
$$\tag 1 x' = x-y-x(x^2+y^2) \\y' = x+y-y(4x^2+y^2)$$
In polar coordinates
$$\tag 2 x^2+y^2 = r^2\\ x = r \cos \theta\\ y = r \sin \theta$$
Differentiating
$$\tag 3 2 x x' + 2 y y' = 2 r r' \implies r r' = x x' + y y' $$
Substituting $(1)$ and $(2)$ into $(3)$
$\begin{align}rr' &= x x' + y y'\\ &= r \cos \theta(r \cos \theta-r \sin \theta-r \cos \theta((r \cos \theta)^2+(r \sin \theta)^2)) +r \sin\theta ( r \cos \theta+r \sin \theta-r \sin \theta (4(r \cos \theta)^2+(r \sin \theta)^2))\\ &= \frac{3}{8} r^4 \cos (4 \theta )-\frac{11 r^4}{8}+r^2\end{align}$
Dividing by $r$
$$r' = \dfrac{r^3}{8}\left(3\cos (4 \theta )-11\right)+r$$
To find the angle
$$\dfrac{r \sin \theta}{r \cos \theta} = \tan \theta = \dfrac{y}{x}$$
Using the quotient rule
$$\theta' = \dfrac{x y' - y x'}{r^2} = ...$$
Can you continue?
You should end up with
$$\theta' = r \left(\sin ^2(\theta )+\cos ^2(\theta )-3 r^2 \sin (\theta ) \cos ^3(\theta )\right) = \dfrac{r^3}{8} \left(-6 \sin (2 \theta )-3 \sin (4 \theta )\right) + r$$
Now you have $r'$ and $\theta'$, please continue.