Consider the system
$\dot{x} = x(1-4x^2-y^2)-\frac{1}{2}y(1+x) $
$\dot{y} = y(1-4x^2-y^2)-2x(1+x) $
Show that origin is an unstable fixed point
I made $\dot{x} = 0$ and $\dot{y}=0$ and $\dot{x} = \dot{y}$ . Then found an equation,$x=0,y=\frac {\sqrt3}{2}$ but when I did y=0, i found imaginary x. Then I did $\frac{\dot{y}}{\dot{x}}$. I cant continue now on.
I am editing, actually adding one more question.I confused because of that
Here is it,
By considering, $\dot{V}$ , wherer $V=(1-4x^2-y^2)^2$, show all trajectories approach the ellipse $4x^2+y^2=1$ as t goes to infinity
Hints:
Note: It looks like you were trying to find all of the fixed points. I am not sure if you were supposed to find all of the fixed (critical) points, but there are four of them, which includes the point $(x,y) = (0,0)$.
The fixed points are:
Here is a phase portrait showing these.
Updates to Question