I consider the a nonlinear system of the following:
\begin{align}\dot x&=x^2 + 3y^2 - 1 \\ \dot y &= -2xy \end{align}
I have to demonstrate that $x^2 + y^2 = 1$ is an invariant set, but not periodic orbits.
My attempt is that:
I first note that this system is Hamiltonian so that I can find a Hamiltonian function which is of the form: $$ H(x,y) = x^2 y + y^3 - y $$
I find that: if $(x,y) \in C = \{(x,y) \in \mathbb{R}^{2} | x^2 + y^2 = 1 \}$, then
$$ H(x,y) \equiv 0 $$
so $C$ is an invariant set.
But I am not really sure how to proceed to show that there exists no periodic orbits. Any hints will be much appreciated. Thanks in advance!.
You have correctly shown that the unit circle is invariant.
To show that it contains no periodic orbits, consider the equilibrium points of the system.