I am asked to find all equilibrium solutions to this system of differential equations:
$$\begin{cases} x ' = x^2 + y^2 - 1 \\ y'= x^2 - y^2 \end{cases} $$
and to determine if they are stable, asymptotically stable or unstable.
I do not know how to proceed, I tried switching to polar coordinates to make the system linear in some way but I did not succeed.
Would I need to use a software to solve this?
Context: I am following the course of Arthur Mattuck (MIT opencourseware), but I can't seem to find these type of systems, or equilibrium points.
We are given:
$$\begin{cases} x ' = x^2 + y^2 - 1 \\ y'= x^2 - y^2 \end{cases} $$
As @RobertLewis has pointed out, we find the equilibrium points $(x,y)$ at the points where we have $x' = y' = 0$. We have
$x^2 + y^2 -1 =0\, \tag{3}$
$x^2 - y^2 = 0. \tag{4}$
From $(4)$, we have $x^2 = y^2$. Substituting this back into $(3)$, yields $x=\pm \dfrac{1}{\sqrt{2}}$. We can substitute this $x$ back into $(4)$, yielding $y = \pm \dfrac{1}{\sqrt{2}}$. Thus, we have a total of critical points as:
$$(x, y) = \left(-\dfrac{1}{\sqrt{2}},-\dfrac{1}{\sqrt{2}}\right), \left(-\dfrac{1}{\sqrt{2}},\dfrac{1}{\sqrt{2}}\right), \left(\dfrac{1}{\sqrt{2}},-\dfrac{1}{\sqrt{2}}\right), \left(\dfrac{1}{\sqrt{2}},\dfrac{1}{\sqrt{2}}\right)$$
You should validate that each of these four points gives you $x'= y' = 0$ by substituting them into the original system.
Your next step is to use linearization, find the Jacobian and evaluate the eigenvalues for those four critical points to determine stability. I am going to let you work that, but here are some nice notes with a summary, the gotchas with linearzation and examples (starts on page $4$). I am also not sure if you discussed nullclines, but the notes have those also and they can be superimposed on the phase portrait. Hint: there are three unstable and one stable critical point.
Here is a phase-portrait for the system showing the four critical points (which can be used to validate your critical points and stability analysis).