I have a doubt that really bothers me. I Always thought that if in ODE I find a conserved quantity whose level sets are compact, than the solutions have to be periodic. However, in the following system of ODE, things seem different: consider $$x'=-xy,\quad y'=x^2;$$
than one can easily check that $F(x, y)=x^2 + y^2$ is a conserved quantity, but since $y'\geq 0$, then $y$ admits limit for $t\to\infty$, and actually $x$ admits it too. So how can these solutions be periodic?
If nothing is wrong, then in general we can't conclude that if level sets are compact solutions are periodic, so, in this case, what conditions could allow us to get to such a conclusion?
Thank you in advance.
Nothing is wrong. Your observation is correct. Only because the ovals of the first integral are compact doesn't mean the solutions in their original parametrization are periodic. Actually, in your example, if you switch to polar coordinates, the system turns into \begin{align} r' &= 0\\ \theta' &= r \cos{\theta} \end{align}
so for any $r_0>0$ the system reduces to $\theta' = r_0 \cos{\theta}$ and thus the solutions are given by $$F\big(\theta(t)\big) = \int_{\theta_0}^{\theta(t)} \frac{d \varphi}{\cos{\varphi}} = r_0(t-t_0)$$ in implicit form. The function $F(\theta)$ has vertical asymptotes at distance $\pi$ from each other, so $\theta(t)$ cannot go from say $0$ to $2\pi$, but will go to infinity before that.
Where is the catch?
Let us look at the planar system \begin{align} x' &= P(x,y)\\ y' &=Q(x,y) \end{align} and let us assume that the smooth function $H(x,y)$ is such that for some regular value $c_0 \in \mathbb{R}$ the level curve $$\Gamma_{c_0} = \big\{(x,y) \in \mathbb{R}^2 \, \, | \,\, H(x,y) = c_0\big\}$$ is a smooth simple closed curve such that any solutions of the system starting on $\Gamma_{c_0}$ always stay on $\Gamma_{c_0}$. This means $\Gamma_{c_0}$ is an invariant curve of the system, equivalently means that the vector field determined by the ODEs is tangent to $\Gamma_{c_0}$. In particular, if $H$ is a first integral of the system, then $\Gamma_{c_0}$ automatically invariant.
Let us rewrite our system as \begin{align} dt &= \frac{dx}{P(x,y)}\\ dt &= \frac{dy}{Q(x,y)} \end{align} Now define the one forms $\omega_1 = \frac{dx}{P(x,y)} \Big| _{\Gamma_{c_0}} $ and $\omega_2 = \frac{dy}{Q(x,y)} \Big| _{\Gamma_{c_0}}$ on $\Gamma_{c_0}$ where possible. Then you see that on the open subset of $\Gamma_{c_0}$ where $P(x,y) \neq 0$ and $Q(x,y) \neq 0$ the two forms coincide, i.e. $\omega_1 = \omega_2 = \omega$. This is because $\Gamma_{c_0}$ is an invariant oval of the differential equations. Now, this means that $\omega$ extends as a smooth one form on the open subset of $\Gamma_{c_0}$ where $P(x,y) \neq 0$ or $Q(x,y) \neq 0$ or both. In particular if this set is the whole $\Gamma_{c_0}$ then $\omega$ is a smooth one form on $\Gamma_{c_0}$. In that case $$\int_{\Gamma_{c_0}} \omega = \int_{\Gamma_{c_0}^P} \frac{dx}{P(x,y)} = \int_{\Gamma_{c_0}^Q} \frac{dy}{Q(x,y)} = T$$ where $\Gamma_{c_0}^P$ is the open portion of $\Gamma_{c_0}$ where $P(x,y) \neq 0$. The same for $\Gamma_{c_0}^Q$. The equalities above are true assuming that $P(x,y) = 0$ and $Q(x,y) = 0$ are sets of measure zero, which is the case of isolated points (a very generic case). Then the value $T$ is the period of a solution on the oval $\Gamma_{c_0}$. And there you have it, if $P$ and $Q$ are not simultaneously zero on $\Gamma_0$ then the solutions on $\Gamma_{c_0}$ are periodic with period $T$.
However, if both $P$ and $Q$ become zero on the oval $\Gamma_{c_0}$ you run into trouble. Does your system fall into that category?
\begin{align} \end{align}
\begin{align} \end{align}