We have the system of differential equations: $$x'=(1+m)y+x(1-(x^2+y^2))(4-(x^2+y^2)),$$ $$y'=-x+y(1-(x^2+y^2))(4-(x^2+y^2)),$$ with $m>0$.
- How do I show that $(0,0)$ is the only (instable) critical point?
- How do I show that there is an $m_0$ such that for $0<m<m_0$ this system has TWO periodic (not-constant) solutions?
What I have done so far: I wrote the sytem above in polar coordinates:
$$r'=mr\sin\theta\cos\theta+r(1-r^2)(4-r^2),$$
$$\theta'=-1-m\sin^2\theta.$$
But I don't know how this can help me.
For question 2. I must use Poincare-Bendixson, but I don't see how exactly.
First of all the proof that the origin is the only critical point can be proved following the comments of @Hans Engler. I will skip the details.
For the second question there are possibly other ways to prove this by I will present one that looks appealing to me. Consider the function $$V:=r^2=x^2+y^2$$ The time derivative of $V$ is given by $$\dot{V}=-2m\: xy+2V(1-V)(4-V)$$ that can be written equivalently as a quadratic form $$\dot{V}=\left[\matrix{x & y}\right]\left[\matrix{2(1-V)(4-V) & -m\\-m & 2(1-V)(4-V)}\right]\left[\matrix{x \\ y}\right]$$ From the above equation $\dot{V}< 0$ iff $$(i)\quad 1<V<4\\(ii)\quad m<2(V-1)(4-V)$$ and $\dot{V}>0$ iff $$(i)\quad 0<V<1\quad or \quad V>4\\(ii)\quad m<2(1-V)(4-V)$$ Note that the maximum value of $2(V-1)(4-V)$ when $V\in(1,4)$ is $4.5$ and is achieved if $V=2.5$. Thus, for $0<m<m_0=4.5$ we have $$\dot{V}<0\text{ whenever } V=2.5$$ Also, if $0<m<m_0=4.5$ then for any $\epsilon\in\left(0,\frac{5-3\sqrt{2}}{2}\right]$ we have that $$\dot{V}>0 \text{ whenever } V=\epsilon$$
This means that if initially $V(0)\in[\epsilon,2.5]$ then $V(t)\in[\epsilon,2.5]$ for all $t\geq 0$. Define now the bounded closed subset of the plane $R:=\{(x,y)|\epsilon\leq V \leq 2.5\}$. Obviously $R$ does not contain fixed points (the origin) and all trajectories starting in $R$ are confined in $R$. Thus, the Poincare-Bendixson theorem can be directly applied to prove the existence of a periodic trajectory inside $R$.