I want to write this ODE system in polar coordinates (r,$\theta$).
$$\dot x =x-y-x^3 $$
$$\dot y = x+y-y^3$$
We use:
$$x = r \cos \theta, y = r \sin \theta, r^2 = x^2 + y^2, \theta = \tan^{-1}\left(\frac{y}{x}\right)$$
From:
$$x^2 + y^2 = r^2 \implies x x' + y y' = r r'$$
We have:
$$\begin{align} x x' + y y' &= x(x-y-x^3) + y(x+y-y^3) \\ &= r \cos \theta(r \cos \theta-r \sin \theta - (r \cos \theta)^3) + r \sin \theta(r \cos \theta + r \sin \theta -(r \sin \theta)^3) \\&= r r'\end{align}$$
We also have:
$$\theta = \tan^{-1}\left(\frac{y}{x}\right) \implies \theta' = \dfrac{xy'-yx'}{r^2}$$
Make the appropriate substitutions, do some trig simplifications and cleanup to derive $r'$ and $\theta'$.
Update
You should end up with:
$$r' = -\frac{1}{4} r \left(r^2 \cos (4 \theta)+3 r^2-4\right)$$
$$\theta' = -\left(\left(r^2-1\right) \cos (2 \theta)+\sin (2 \theta)\right)$$
So $r$ and $\theta$ are function of $t$, let $x=r\cos\theta$ and $y=r\sin \theta$, then you will have
$$r'\cos \theta-r\sin\theta \times (\theta)'=r\cos\theta-r\sin \theta-(r\cos\theta)^3;$$
$$r'\sin \theta+r\cos\theta \times (\theta)'=r\cos\theta+r\sin \theta-(r\sin\theta)^3$$
Copyright © 2021 JogjaFile Inc.
We use:
$$x = r \cos \theta, y = r \sin \theta, r^2 = x^2 + y^2, \theta = \tan^{-1}\left(\frac{y}{x}\right)$$
From:
$$x^2 + y^2 = r^2 \implies x x' + y y' = r r'$$
We have:
$$\begin{align} x x' + y y' &= x(x-y-x^3) + y(x+y-y^3) \\ &= r \cos \theta(r \cos \theta-r \sin \theta - (r \cos \theta)^3) + r \sin \theta(r \cos \theta + r \sin \theta -(r \sin \theta)^3) \\&= r r'\end{align}$$
We also have:
$$\theta = \tan^{-1}\left(\frac{y}{x}\right) \implies \theta' = \dfrac{xy'-yx'}{r^2}$$
Make the appropriate substitutions, do some trig simplifications and cleanup to derive $r'$ and $\theta'$.
Update
You should end up with:
$$r' = -\frac{1}{4} r \left(r^2 \cos (4 \theta)+3 r^2-4\right)$$
$$\theta' = -\left(\left(r^2-1\right) \cos (2 \theta)+\sin (2 \theta)\right)$$