The full question reads as follows:
Consider the predator-prey model
$$\dot{x} = x \left(6-x-\frac{3y}{1+x} \right), \dot{y} = y(x-2)$$
Assume that all positive solutions are bounded. (a) Find all critical points and determine their local stability. (b) Show that this system has a limit cycle in the first quadrant.
I have finished part (a) and found that $(0,0)$ is a saddle, $(6,0)$ is a saddle, and $(2,4)$ is an unstable focus. I have attempted to do part (b) using some phase plane analysis, but I am looking for a more definite way to prove this. Thanks in advance for any help!
Assume that all solutions starting in $\mathbb{R}^2_{+} := \{\, (x, y) : x \ge 0,\ y \ge 0 \,\}$ are bounded for $t > 0$. It follows then that the domain of any such solution contains $[0, \infty)$, and that the $\omega$-limit set is compact and nonempty.
Let $L$ stand for the $\omega$-limit set of some point, $(x_0, y_0)$, sufficiently close to the unstable focus $(2,4)$. By the Poincaré–Bendixson theorem, $L$ is either a limit cycle, or a heteroclinic cycle, (EDIT: or a homoclinic loop), or an equilibrium.
$L$ cannot be an equilibrium. Indeed, there are three possibilities: either $L = \{(0, 0)\}$, or $L = \{(6,0)\}$, or else $L = \{(2, 4)\}$. In the first two cases, the equilibria are saddles, so $(x_0, y_0)$ must belong to the stable manifold, which means either $\{\, (0, y) : y > 0 \,\}$ or $\{\, (x, 0) : x \in (0, 6) \cup (6, \infty) \,\}$. In the third case, the only point whose $\omega$-limit set is $\{(2,4)\}$ is $(2,4)$ itself.
We proceed now to excluding the case of heteroclinic cycle (EDIT: or homoclinic loop). But the only possible connection is from $(0, 0)$ to $(6,0)$, so there are no heteroclinic cycles.
Consequently, $L$ is a limit cycle surrounding $(2,4)$.
Below is a sketch of the phase portrait:
Incidentally, it is not so easy to show that all solutions starting in $\mathbb{R}^2_{+}$ are bounded for positive times. The OP gave that as an assumption, but it should follow somehow from the form of the system.