Consider the autonomous dynamical system: $$ \left\{ \begin{array}{l} \dot{x}=x+y-x^3-\alpha xy^2\\ \dot{y}=y-x-x^2y-y^3 \end{array} \right. $$ Prove that $(0,0)$ is the unique critical point if $\alpha\in(0,2)$.
My work: I tried to use polar coordinates but system with the change is not easier to work with. I also works algebraically with the system with variables $x,y$ and the parameter $\alpha$, but it didn't come to anything interesting either.
Hint.
Assuming that the dynamic system is
$$ \left\{ \begin{array}{l} \dot x=x+y-x^3-\alpha xy^2\\ \dot y=y-x-x^2y-y^3 \end{array} \right. $$
we have
$$ \left\{ \begin{array}{l} x\dot x=x^2+y x-x^4-\alpha x^2y^2\\ y\dot y=y^2-x y-x^2y^2-y^4 \end{array} \right. $$
and after addition
$$ \frac 12(x^2+y^2)'=x^2+y^2-(x^4+(1+\alpha)x^2y^2+y^4) $$
now if $x^2+y^2-(x^4+(1+\alpha)x^2y^2+y^4) < 0$ the equilibrium point is $(0,0)$ but calculating the hessian at $(0,0)$ we found that
$$ H = \left(\matrix{2 & 0\\ 0 & 2}\right) $$
which is positive definite hence no equilibrium point at $(0,0)$. At $(0,0)$ we have a source and for $a>0$ a limit cycle around $(0,0)$.
Attached a stream plot for $\alpha = 1$
NOTE
In polar coordinates, making $\{x = \rho \cos\theta, y = \rho\sin\theta\}$ we have
$$ \frac{1}{8} \rho ^2 \left((\alpha-1) \rho ^2 \cos (4 \theta )-(\alpha+7) \rho ^2+8\right) = 0 $$
and solving for $\rho$ we have
$$ \rho = \cases{0\\ \frac{2 \sqrt{2}}{\sqrt{\alpha+7-(\alpha-1) \cos (4 \theta )}}} $$
and for $\alpha = 1$ we have $\rho = 1$ as depicted in the previous plot.