Using polar co-ordinates to solve the dynamical system: $$x^\prime = x-y-x^{3}$$
$$y^\prime = x+y-y^{3}$$
I have arrived at: \begin{align} r' &= r-r^3(cos^4\theta+sin^4\theta)\\ \theta' &= 1+\frac{r^2}{4}sin4\theta \end{align}
However I am now stuck on how to proceed in finding the fixed point. I know that I need to make both $r'$ and $\theta'$ equal to zero but can't find the breakthrough as to what values $r$ and $\theta$ would need to take to achieve this. Any help with this is appreciated.
It will not be a fixed point, but a limiting cycle. This takes the form of a closed curve whose distance from the origin must be proved less than $2$ at all points.
You just keep going from your expression for $r'$. Dividing out a factor of $r$ gives for $r\ge 2$:
$r'/r\le1-4(\cos^4\theta+\sin^4\theta)$
And then
$\cos^4\theta+\sin^4\theta=(1/2)[(\cos^4\theta+2\cos^2\theta\sin^2\theta+\sin^4\theta)+(\cos^4\theta-2\cos^2\theta\sin^2\theta+\sin^4\theta)]$
$=(1/2)[(\color{blue}{\cos^2\theta+\sin^2\theta})^2+(\cos^2\theta-\sin^2\theta)^2]$
Given the value of the blue expression we conclude that $r'/r\le-1$ for $r\ge2$ with obvious implications for the motion of the object in that range.
A little homework for you: using the identity given for $\cos^4\theta+\sin^4\theta$, prove that the limit cycle actually lies in $1\le r\le\sqrt2$.