The system is the following: $\left\{\begin{matrix} x'=&2x-x^5-xy^4 \\ y'=&y-y^3-x^2y \end{matrix}\right.$.
What I did was to find singular points in the system: from which I got the following singular points: $(0,0); (0,1);(0, -1);(\sqrt[4]{2},0);(\sqrt[4]{2},0)$. Then, I have classified them if they are attractors, chair, etc. But I do not know how to prove that the system does not have periodic orbits. Can someone guide me? Please. Thanks in advance. Best regards.
It can be observed that the lines $x= 0$ and $y=0$ are invariant sets of the system: $$ \dot x\big|_{x=0}=0,\quad \dot y\big|_{y=0}=0. $$ It means that no solution can enter or leave these lines.
In fact, the phase portrait of the system looks as follows:
(the fixed points are denoted by asterisks).
We know that any closed orbit on a plane must enclose at least one fixed point. But for this system, this is impossible. Any closed orbit here must traverse the "red" solutions, which violates the uniqueness and existence theorem.