The full problem reads:
Show that the system with $\dot{x} = y$ and $\dot{y} = -2x-y-3x^4+y^2$ has no limit cycle in $\mathbb{R}^2$. (HINT: a function in the form $\mathrm{e}^{\beta x}$ might be useful)
I know that I need to use Dulac's Criteria here, but I can't find the correct $\beta$ to go along with the hint. Any advice is welcome!
Thanks!
For any planar system of the form
$\dot x = X(x, y), \tag 1$
$\dot y = Y(x, y), \tag 2$
the Bendixson-Dulac criterion states that there are no periodic orbits in a region $\Omega$ where
$\nabla \cdot (\phi(X, Y)) = \nabla \cdot (\phi X, \phi Y) \ne 0, \tag 3$
where $\phi(x, y)$ is some differentiable scalar function on $\Omega$. For the present system, we may take $\Omega = \Bbb R^2$ and
$\dot x = y = X(x, y), \tag 4$
$\dot y = -2x - y - 3x^4 + y^2 = Y(x, y); \tag 5$
if we set
$\phi(x, y) = e^{\beta x}, \tag 6$
then we calculate:
$\nabla \cdot (\phi(X,Y)) = \dfrac{\partial (\phi X)}{\partial x} + \dfrac{\partial (\phi Y)}{\partial y} = \dfrac{\partial (e^{\beta x}y)}{\partial x} + \dfrac{\partial (e^{\beta x}(-2x - y - 3x^4 + y^2))}{\partial y}$ $= \beta e^{\beta x}y + (e^{\beta x}(-1 + 2y)) = e^{\beta x}(\beta y - 1 + 2y) = e^{\beta x}((\beta + 2)y - 1); \tag 7$
we observe that the function $e^{\beta x}((\beta + 2)y - 1)$ occurring on the right-hand side of (7) takes on the value zero precisely at that $y_0$ for which
$(\beta + 2)y_0 - 1 = 0, \tag 8$
that is, along the line
$y = y_0 = \dfrac{1}{\beta + 2}, \tag 9$
in the $xy$-plane. If we choose
$\beta = \epsilon - 2, \tag{10}$
for some real $\epsilon$, then since
$\epsilon = \beta + 2, \tag{11}$
we see that
$y_0(\epsilon) = \dfrac{1}{\beta + 2} = \dfrac{1}{\epsilon}, \tag{12}$
and we further observe that by choosing $\epsilon > 0$ sufficiently small, not only may we ensure that $y_0(\epsilon)$ is arbitraritly large, but also that the function
$(\beta + 2)y - 1 = \epsilon y - 1 \tag{13}$
increases with increasing $y$; thus it is positive for $y > y_0$ and negative for $y < y_0$.
Now if we had a periodic orbit, it would form the boundary of a compact subset $K \subset \Bbb R^2$ upon which the function $y$ is continuous, hence bounded above by some
$y_M = \sup \{y \mid y \in K \}; \tag{14}$
thus for suffiiently small $\beta + 2 = \epsilon > 0$ we have
$y_M < y_0; \tag{15}$
that is, on $K$,
$(\beta + 2)y - 1 < 0 \Longrightarrow e^{\beta x}((\beta + 2)y - 1) < 0; \tag{16}$
but now by Bendixson-Dulac, via (7), this contradicts the existence of a periodic trajectory of the system (1)-(2); thus in particular this system cannot have a limit cycle in $\Bbb R^2$.