Given the following dynamical system $$\tag{1}\begin{cases}\label{eq_1} \dot x = x^2 +3y^2-2xy-1 \\ \dot y = 3x^2+y^2-2xy-1 \end{cases}$$ Find a constant of motion and sketch a phase diagram.
I know that $\Phi(x,y)$ a dynamical variable is a constant of motion $\iff \Phi(x(t),y(t)) = c$ a constant for every movement $(x(t),y(t))$ system's solution. This is equivalent to $$\dot x\frac{\partial\Phi}{\partial x} + \dot y \frac{\partial \Phi}{\partial y} = 0 $$ Then with this last equation I can say that a possibile constant of motion is $$\frac{\partial\Phi}{\partial x} = \dot y, \frac{\partial\Phi}{\partial y} = -\dot x \\ \Phi(x,y) = (x-y)(x^2+y^2-1)$$
I want to investigate equilibrium points, with my analysis using the condition $x=y$ and $x=-y$ with the system $\eqref{eq_2}$ $$\tag{2}\begin{cases}\label{eq_2} x^2 +3y^2-2xy-1 = 0 \\ 3x^2+y^2-2xy-1 = 0 \end{cases} $$ Equilibrium points are $A= (\sqrt{2}/2,\sqrt{2}/2), B = (-\sqrt{2}/2,-\sqrt{2}/2), C = (\sqrt{6}/6, -\sqrt{6}/6), D = (-\sqrt{6}/6,\sqrt{6}/6)$ and studying the Jacobian of $\eqref{eq_1}$ I think they are all saddles.
Now comes the problems because looking at online dynamical simulator $C,D$ does not appear in the phase diagram that looks like below
What am I missing thank you so much for the help!
Another mistake is indeed one direction corresponding to the positive eigenvalue should expand and the other one should contract. It seems looking at the picture that this behaviour does not appear...
I agree with your critical points.
For one of the ellipses, the critical point $(\frac{\sqrt{6}}{6},-\frac{\sqrt{6}}{6})$, produces a Jacobian
$$\left( \begin{array}{cc} 2 \sqrt{\frac{2}{3}} & -\sqrt{6}-\sqrt{\frac{2}{3}} \\ \sqrt{6}+\sqrt{\frac{2}{3}} & -2 \sqrt{\frac{2}{3}} \\ \end{array} \right)$$
The eigenvalues for this example are purely imaginary
$$\left\{2 i \sqrt{2},-2 i \sqrt{2}\right\}$$
A phase portrait with the four critical points in red is
We can add a lot more detail as
Observations from the analysis and phase portraits:
Two eigenvalues are opposite sign which are saddle points (the critical points with $\sqrt{2}$ terms).
Two eigenvalues are purely imaginary which are ellipses (the critical points with $\sqrt{6}$ terms).
You can try using this phase plotter or others online to plot phase portraits and then just click on the screen for a bunch of initial points and compare to yours.
These are also a nice set of notes.