Given the following differential equation
$$\ddot x + (x^2+\dot x^2-1) \dot x + x = 0$$
find the polar equations of motion in state space.
This is for my classical mechanics course, but it seems more mathematics than physics, which is why I am asking this here. I am totally lost. I thought about using a substitution $y = \dot x$, which results in
$$\dot y = (1-x^2)y-y^3$$
which doesn't seem oscillatory. I also see that it is very similar in form to the van der Pol oscillator, but there are additional terms. So I am just looking for a hint on how to go about this. Thanks.


Great question, interesting problem, one of my favorites, wish I could upvote the question more than once!
So the equation of interest is
$\ddot x+(x^2+\dot x^2-1)\dot x + x=0. \tag{1}$
Well, we want a polar expression for (1). So we begin by making the two-dimensional, planar aspects of (1) obvious, since polars can't really be applied in any nice way to an equation on the line; this transformation is affected by introducing a new variable $y$, and setting $y = \dot x$, so:
$\dot x = y, \tag{2}$
then substituting $y$ from (2) into (1) we obtain
$\dot y = - (x^2 + y^2 - 1)y - x. \tag{3}$
Taken together, (2) and (3) form a planar system, often called the phase representation of the equation (1); a point in the $x$-$y$ plane now may be interpreted as a complete state of (1), since it contains all the information needed to determine the ensuing trajectory uniquely, i.e., position and velocity. We cast (2), (3) in polar form by setting
$x = r \cos \theta, \tag{4}$
$y = r \sin \theta, \tag{5}$
from which readily follows
$\dot x = \dot r \cos \theta -r \dot \theta \sin \theta, \tag{6}$
$\dot y =\dot r \sin \theta + r\dot \theta \cos \theta. \tag{7}$
We now insert (4)-(7) into (2), (3) and, as the old song says, "boil that cabbage down", so to speak, by cooking things on the stove of algebraic manipulation:
$\dot r \cos \theta -r \dot \theta \sin \theta = r\sin \theta, \tag{8}$
$\dot r \sin \theta + r\dot \theta \cos \theta = - (r^2 \cos^2 \theta + r^2 \sin^2 \theta - 1) r \sin \theta - r \cos \theta;\tag{9}$
now, as Bob Marley sang, we Stir It Up; so, pretty baby, we re-write (8), (9) a little bit. Notice that, in terms of the vector $(\dot r, \dot \theta)^T$ the combined lefts of these two equations can be written in matrix-vector form, whence we arrive at
$\begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix} \begin{pmatrix} \dot r \\ \dot \theta \end{pmatrix} = \begin{pmatrix} r \sin \theta \\ -(r^2 - 1)r \sin \theta -r \cos \theta \end{pmatrix}. \tag{10}$
Now as long as $r \ne 0$, the matrix on the left-hand side of (10) is nonsingular; indeed its determinant is $r$:
$\det \begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix} = r(\cos^2 \theta + \sin^2 \theta) = r; \tag{11}$
thus it may be simply inverted; indeed, a standard calculation yields
$\begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix}^{-1} = \frac{1}{r} \begin{bmatrix} r\cos \theta & r\sin \theta \\ -\sin \theta & \cos \theta \end{bmatrix}; \tag{12}$
keep stirring, keep stirring, keep boiling: multiply (10) by (12) and grind; we obtain
$\begin{pmatrix} \dot r \\ \dot \theta \end{pmatrix} = \begin{pmatrix} -r(r^2 - 1) \sin^2 \theta \\ -(r^2 -1)(\cos \theta)(\sin \theta) -1 \end{pmatrix}, \tag{13}$
the equation (1) in polar form may thus be written out in non-vector form as
$\dot r = -r(r^2 - 1) \sin^2 \theta, \tag{14}$
$\dot \theta = -(r^2 -1)(\cos \theta)(\sin \theta) -1 = -\frac{1}{2}(r^2 -1)\sin 2 \theta -1. \tag{15}$
There! Cabbage ready! It is worth noting that (14) implies, as long as $\sin^2 \theta \ne 0$, $r$ increases as long as $0 < r < 1$ and decreases when $r > 1$; when $r = 1$ we have
$\dot r = 0, \tag{16}$
and
$\dot \theta = -1; \tag{17}$
the circle $r = 1$ is apparently a stable, closed, periodic orbit of period $2\pi$, rotating in the clockwise direction. In the event $\sin^2 \theta = 0$, we have that $\theta = m\pi$ for some integer $m$; then $\sin 2\theta = 0$ as well, so that from (15) $\dot \theta = -1$; the condition $\dot r = 0$ is thus seen to be an "instantaneous" phenomenon; $r$ will continue to increase/decrease along a trajectory except at these particular values of $\theta$, with $r \to 1$. Properties of solutions near points where $\dot \theta = 0$ may be investigated using (15); I leave such questions to my readers. Other interesting features of the flow may be derived from examination of (15), (16); but my night job beckons, and so I must bid you all adieu.
Hope this helps. Cheerio,
and as always,
Fiat Lux!!!