I have the linear system:
$$\begin{cases}\dot{x}=y\\ \dot{y}=-ay+x-x^3\end{cases}$$
where $a\geq 0$.
I want to prove that this dynamical system has two minimum.
I found the 3 equilibrium points $(0,0)$, $(1,0)$, $(-1,0)$.
I calculated the jacobian and i don't know how to proceed. I have read something about minimum points when the Jacobian is positive definitive but these matrices are not.
As @GReyes proposed you need to linearize the system at the equilibrium points $\boldsymbol{x}_\text{eq}=[x_\text{eq},y_\text{eq}]^T$, which are $[0,0]^T$ and $[\pm 1,0]^T$.
If we write the system in vector form, by using $\boldsymbol{x}=[x,y]^T$ and $\boldsymbol{f}=[f_1(\boldsymbol{x}),f_2(\boldsymbol{x})]^T$ $$\dot{\boldsymbol{x}}=\boldsymbol{f}(\boldsymbol{x})$$ we can simply get the linarized system matrix $\boldsymbol{A}$ by
$$\boldsymbol{A} = \left.\dfrac{\partial \boldsymbol{f}}{\partial \boldsymbol{x}}\right|_{\boldsymbol{x}=\boldsymbol{x}_\text{eq}}$$
Written out for our problem this is given by
$$\boldsymbol{A}=\begin{bmatrix} \dfrac{\partial f_1}{\partial x}& \dfrac{\partial f_1}{\partial y}\\ \dfrac{\partial f_2}{\partial x}& \dfrac{\partial f_2}{\partial y} \end{bmatrix}_{\boldsymbol{x}=\boldsymbol{x}_\text{eq}}$$ $$=\begin{bmatrix}0 & 1\\1-3x^2_\text{eq} & -a\end{bmatrix}.$$
The eigenvalues of this system are given by the characteristic polynomial
$$\chi(s)=s^2+as+(3x^2_\text{eq}-1).$$
If $a>0$ and $3x^2_\text{eq}-1>0$ are fulfilled the system is Hurwitz, hence the equilibrium point is asymptotically stable. You can quickly check these properties for the given equilibrium points. Only the origin is unstable but all other equilibrium points are asymptotically stable in a neighborhood of the equilibrium points.
As @Kwin van der Veen proposed. The phase diagram looks like a landscape with hills (unstable equilibrium points) and valleys (asymptotically stable equilibrium points). If you imagine a ball starting somewhere (initial conditions) in this landscape the ball will converge to one of the asymptotically stable equilibrium points (we exclude the trivial origin as a starting point), by following the dynamics of the system (given by the arrows in the phase diagram provided by @Kwin van der Veen).
The hard part is actually the case $a=0$. For the equilibrium point $[0,0]^T$ you can still use the linearization theorem by Lyapunov. But for the other equilibrium points, you will need to find a Lyapunov function to show stability.
EDIT: I just couldn't resist :D. In order to show the stability of the system at the equilibrium points $[0,\pm 1]^T$ we divide both equations to obtain
$$dy/dx = \dfrac{x-x^3}{y} \implies 1/2y^2-1/2x^2+1/4x^4-c=0.$$
Now, we define $\Phi(x,y)= 1/2y^2-1/2x^2+1/4x^4$ and evaluate this function at the equilibrium points $x_\text{eq}=\pm 1$ to get $\Phi_\text{eq}=-1/4$. Now, we construct a Lyapunov function candidate
$$V(x,y)=\Phi(x,y)-\Phi_\text{eq}=1/2y^2-1/2x^2+1/4x^4+1/4$$ $$=1/4\left[(1-x^2)^2+2y^2\right].$$
You can see that the function $V$ is positive definite in the vicinity of the equilibrium points $[\pm 1,0]^T$ and $V=0$ for the equilibrium points. It is clear that $\dot{V}=0$. Because $\Phi(x,y)$ and $\Phi_\text{eq}$ are constants. You can also derive this by differentiating $V$ and using the differential equations for $a=0$. Hence, we can conclude that the equilibrium points are stable.