Given the system:
$\dot x$ = $y$, $\dot y$ = $\epsilon x^3 - x$,
equilibria points are (0,0), $(\frac{1}{\sqrt \epsilon}, 0) $, $(-\frac{1}{\sqrt \epsilon}, 0) $
Thus, the jacobian matrix is:
$$ J = \begin{bmatrix} 0 &1 \\ 3\epsilon x^2 - 1 & 0 \end{bmatrix} $$
At the origin (0,0)
$$ J = \begin{bmatrix} 0 &1 \\ - 1 & 0 \end{bmatrix} $$
This is a degenerate case, i.e. a centre, so how can you determine the stability of this equilibrium point by converting to polar coordinates?
This system has a first integral $$ V(x,y)=y^2+x^2-\fracϵ2x^4. $$ This leads to concentric closed level curves around $(0,0)$, and the solutions of the system follow these curves.