Consider the system
$$x' = 3xy^2-x^2y \\ y' = 5x^2y - xy^2$$
Show that the system has no periodic solutions.
This is a tricky example. Linearization leads nowhere and I'm having a hard time constructing a Lyapunov function that does the trick. $V = 1/2(x^2+y^2)$ gives
$$V'(x,y) = 3x^2y^2-x^3y +5x^2y^2 -xy^3 = 8x^2y^2 -xy(x^2+y^2))$$
But this doesn't tell us much nice things about the origin. If anything, it looks as though the origin is repelling since small perturbations gives us that the $8x^2y^2$ term dominates the minus term. Maybe it's possible to show that there are no elliptical orbits somehow, but that doesn't exclude other, more exotic, periodic trajectories.
How to proceed...?
The title is misleading: it is the Bendixson's criterion what must be used that states that for the system $$\begin{align} x'&=f(x,y)\\y'&=g(x,y) \end{align} $$ if $\frac{\partial f}{\partial x}+\frac{\partial g}{\partial y}\neq 0$ in a simply connected region $R$, then the system has no closed trajectory inside $R$. (See e.g. http://math.mit.edu/~jorloff/suppnotes/suppnotes03/lc.pdf for a proof)
Now, in your case, first of all, note that the axes are invariant, so no closed trajectory can touch one of them. Then, inside any of the quadrants, compute: $$\begin{aligned} \frac{\partial f}{\partial x}+\frac{\partial g}{\partial y} &= 3y^2-2xy+5x^2-2xy =3\left(y^2-\frac{4}{3}xy+\frac{5}{3}x^2\right) \\ &= 3 \left(\left(y-\frac{2}{3}x\right)^2+\frac{5}{2}x^2\right)>0 \end{aligned}$$ since any (strict) quadrant excludes the origin, and the conclusion results from Bendixson's criterion.