I am given a system
$\dot x = f(x,y) = (x^2 + y^2)(x^3 + y^2x -2y - x) \\ \dot y = g(x,y) = (x^2 + y^2)(y^3 + x^2y +2x - y) $
and I am asked if the fixed point at $(0,0)$ is hyperbolic or asymptotically stable. I am also asked whether the system is a Hamiltonian.
So I know that a hyperbolic fixed point is a fixed point that does not have a centre manifold, so this makes me think that it is a hyperbolic fixed point but I'm not really sure.
Then onto asymptotically stable, my understanding of this is that we require the system to be both Liapounov stable and quasi-asymptotically stable, but I really dont know how to go about this.
Finally about it being a Hamiltonian system. I know in order to be a Hamiltonian the system can be written in the form
$H(q_i,p_i)$ such that $\dot q_i= \frac{\partial H}{\partial p_i} \\ \dot p_i= -\frac{\partial H}{\partial q_i}$
But I really dont know how to do this for this DS.
Also i it helps I converted the DS into polar coords, giving me
$\dot r = r^5 - r^3 \\ \dot \theta = 2r^2$
This is about the stability part:
Asymptotic stability of the equilibrium means that it is stable and attractive. The origin $(0,0)$ is an equilibrium. Consider the Lyapunov function candidate $V(x,y) = \frac{1}{2}x^2 + \frac{1}{2}y^2$. The derivative along the trajectories of $\dot x = f(x,y)$ and $\dot y = g(x,y)$ yields
\begin{align} \dot V = x\,\dot x + y\,\dot y &= x(x^2+y^2)(x^3+y^2x-2y-x) + y(x^2+y^2)(y^3+x^2y+2x-y) \\ & = (x^2+y^2)(x^4+y^2x^2-2yx-x^2) + (x^2+y^2)(y^4+x^2y^2+2xy-y^2) \\ & = (x^2+y^2)(x^4+y^4) + (x^2+y^2)(2y^2x^2) - (x^2+y^2)^2 \\ & = (x^2+y^2)\left((x^4+y^4) +(2y^2x^2) \right) - (x^2+y^2)^2 \\ & = (x^2+y^2)(x^2+y^2)^2 - (x^2+y^2)^2 \end{align} Insert $2V = x^2+y^2$: \begin{align} \dot V & = 8\,V^3 - 4 V^2 = -8\,V^2\left(\frac{1}{2}-V\right) \end{align} (Note: this is qualitatively similar to that what could be derived from the representation in polar coordinates)
Accordingly, $\dot V$ is only negative in the set $\{(x,y)\in\mathbb R^2: x^2+y^2<1 \}$ which contains the origin. That is, the origin is (locally) asymptotically stable. The unit disc defines also the region of attraction.