Consider the IVP for the $2$-dimensional dynamical system ($X=[0, \infty )^2$) $$\dot{x_1}=a-x_1-\frac{4x_1x_2}{1+x_1^2}$$ $$\dot{x_2}=bx_1 \bigg( 1- \frac{x_2}{1+x_1^2} \bigg)$$ for all $t \in I$, and $\textbf x (0)=(x_{10}, x_{20}) \in [0, \infty )^2$, with $a,b >0$.
Show that for certain $a$, $b$, and $c$, there exists an invariant region $$\{f(x_1, x_2) \in X : 0 < x_1 < a, \, \, 0 < x_2 < c \}$$ that contains a periodic orbit for the dynamical system.
My attempt:
So we have a rectangle and we have to choose $a,b,c$ such that the flow always remains within the rectangle. Let's consider each edge and corner of the rectangle. Imagine our graph to be $x_1$ as the horizontal axis and $x_2$ as the vertical.
At corner $(0,0)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.
At corner $(0,c)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.
At line $x_1=0$ so on points $(0,l)$ with $l \in (0,c)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.
At corner $(a,c)$: $$\dot x_1 =-\frac{4ac}{1+a^2}<0, \, \, \forall a,c$$
$$\dot x_2 =ba \bigg( 1- \frac{c}{1+a^2} \bigg)$$ with $ba>0, \, \, \forall a,b$ but we need the sign of $\dot x_2$ to be $\leq 0$ and this is dependent on the sign of the following (so we WANT the following): $$ 1- \frac{c}{1+a^2} \leq 0$$ so then the flow at this corner will go negative $x_1$ direction or down within the region. Either way, flow will still be in the region.
At line $x_2=0$ so on points $(q,0)$ with $q \in (0,a)$: $\dot x_1 =a-q >0, \, \, \forall a,q$, and $\dot x_2 =bq>0$. So the flow is directed in the region.
At line $x_1=a$ so on points $(a,p)$ with $p \in (0,c)$: $$\dot x_1 = - \frac{4ap}{1+a^2}<0, \, \, \forall a,p$$ $$\dot x_2 =ba \bigg( 1- \frac{p}{1+a^2} \bigg)$$ We are unsure on the sign of this but regardless the flow will remain in the region since $\dot x_1 <0$.
At line $\dot x_2 =c$, so on points $(t,c)$ with $t \in (0,a)$: $$\dot x_1 = a-t -\frac{4tc}{1+t^2}<0 $$
$$\dot x_2 =bt \bigg( 1- \frac{c}{1+t^2} \bigg)$$ We are unsure on the signs of both of these but we WANT $\dot x_2 \leq 0$ and then regardless of the sign of $\dot x_1$, the flow will remain in the region.
So we need the flowing inequalities to hold: $$1- \frac{c}{1+a^2} \leq 0$$ $$1- \frac{c}{1+t^2} \leq 0$$ Choose $c=66$ and $a=8$ and then they hold.
Next we need our fixed point to be out of the region and this happens when $$1 + \frac{a^2}{25} < c$$ and this is true for our chosen values of $a$ and $c$. Hence by Poincare-Bendixon theorem, there exists a periodic orbit.
IS THIS CORRECT? PLEASE COMMENT!!
I think we have to use this:
Poincare-Bendixon: Consider $\dot x = f(x)$ with $f \in C^1(X)$ (continuous on first differential) and $X \subset \mathbb R^2$. If the forward orbit, $O^+(x_0)$ (for $x_0 > X$) is contained within a closed bounded set $D$ containing no fixed points, then $\omega (x_0)$ contains a periodic orbit.
Generalized Poincare-Bendixon: Let the hypothesis of the above hold, except now, let $D$ contain fixed points $p_1, ..., p_N$. Then $\omega (x_0)$ is one of the following;
1) a fixed point $p_i$.
2) a periodic orbit.
3) a finite number of fixed points $p_i$ , and a countable number of homoclinic/ heteroclinic orbits between $p_i$ and $p_j$.
Hint:
Find the directions of the vector field along the sides of the rectangle $[0,a]\times[0,c]$, for some carefully chosen $c$. For example, along the side $x_1=0$ you have $\dot x_1=a$ and $\dot x_2=0$, showing that the vector field points inside the rectangle. Similarly, for $x_2=0$ you have $\dot x_1=a-x_1$ and $\dot x_2=bx_1$, and again the vector field points inside the rectangle (you see the pattern).
After that study the vector field in a neighborhood of the equilibrium point.
A minor comment about the wording: you should really write "forward invariant" instead of "invariant", and "equilibrium point" instead of "fixed point".