The Question:
Consider the system
$$\frac{dx}{dt}=x(2-x)-y \qquad \frac{dy}{dt} = y-kx$$
where $k>2$ is a constant.
Show that there are no non-trivial closed trajectories in the quadrant $x,y≥0$ using the Bendixson-Dulac Theorem.
Bendixson-Dulac Theorem:
For the system
$$\frac{dx}{dt} = X(x,y) \qquad \frac{dy}{dt} = Y(x,y)$$
where $X,Y \in C^1$, if there exists $\phi = \phi(x,y) \in C^1$ such that
$$\frac{\partial}{\partial x} \big(\phi X \big)+\frac{\partial}{\partial y} \big (\phi Y \big) >0$$
everywhere in a simply connected region $R$, then the system has no non-trivial closed trajectories lying entirely in $R$.
My Attempt:
So I considered a general (continuously differentiable) function $\phi = \phi (x,y)$. Then
\begin{align} & \frac{\partial}{\partial x} \big(\phi X \big)+\frac{\partial}{\partial y} \big (\phi Y \big) \\ = & (3\phi) + \bigg(-2\phi +2\frac{\partial \phi}{\partial x}-k\frac{\partial \phi}{\partial y} \bigg) x + \bigg(-\frac{\partial \phi}{\partial x}+\frac{\partial \phi}{\partial y} \bigg) y + \bigg(-\frac{\partial \phi}{\partial x} \bigg)x^2 \end{align}
Suppose we insist that each of the four functions in brackets must be everywhere positive in the quadrant $x,y≥0$.
Then $\phi>0$ from the first term.
$\dfrac{\partial \phi}{\partial x}<0$ from the last term.
$\dfrac{\partial \phi}{\partial y}>0$ from the third term.
But this would imply that everything in the second bracket is negative, so this does not work.
And I am stuck. Any hints? Why does it even matter that $k>2$?
OK, I have figured out the answer to my own question.
Basically, what was wrong in my reasoning above, is that if $-\dfrac{\partial \phi}{\partial x}$ is already positive, then $\dfrac{\partial \phi}{\partial y}$ can be negative in the third bracket, so that $-k\dfrac{\partial \phi}{\partial y}$ becomes a positive quantity in the second bracket with which we can use to balance out the negative terms.
With this idea in mind, I set the quantity in the third bracket to $0$, i.e.
$$-\frac{\partial \phi}{\partial x}+\frac{\partial \phi}{\partial y}=0 \implies \frac{\partial \phi}{\partial x}=\frac{\partial \phi}{\partial y}$$
Solving this via separation of variables, we get
$$\phi (x,y) = Ae^{c(x+y)}$$
The constant $A$ is not gonna help us in any way, and we are fine as long as $A>0$, so lets just set $A=1$. Next, looking at the fourth bracket,
$$-\frac{\partial \phi}{\partial x}>0 \implies -ce^{c(x+y)}>0 \implies c<0$$
so $c$ needs to be a negative constant.
Finally, in the second bracket, we have
$$ -2\phi+2\frac{\partial \phi}{\partial x}-k\frac{\partial \phi}{\partial y} = (-2+2c-kc)e^{c(x+y)}$$
and this is positive, as long as $c<\dfrac{2}{2-k}$ which, thankfully, does not contradict with the requirement that $c<0$ (since $k>2$). Indeed, if we choose $c=\dfrac{2}{2-k}$, then $\phi = \exp \Big(\dfrac{2}{2-k} \Big)$
$$\frac{\partial}{\partial x}\big(\phi X \big)+\frac{\partial}{\partial y}\big(\phi Y \big) = \cdots = \bigg(3+\frac{2}{k-2}x^2 \bigg)\exp{\frac{2(x+y)}{2-k}}$$
which is in fact positive everywhere in $\Bbb R^2$, assuming $k>2$.