My system: $$\frac{\mathrm{dx} }{\mathrm{d} t}=-ax^2+y^2-\gamma z$$ $$\frac{\mathrm{dy} }{\mathrm{d} t}=- h(y)-\beta y $$ $$\frac{\mathrm{dz} }{\mathrm{d} t}=x+h(y)-\beta z $$
where $h$ is the piecewise function
$$h(y)=\alpha(m_1 y+\frac{1}{2}(m_0 - m_1)(|y+1|-|y-1|)).$$
$$\frac{\mathrm{d} h}{\mathrm{d} y}=\alpha m_1+0.5(m_0-m_1)(\frac{y+1}{|y+1|}-\frac{y-1}{|y-1|})$$, is this right? Upon solving, I get the Jacobian $J$
$$J= \left( \begin{array}{ccc} -2ax & 2y & -\gamma \\ 0 & \alpha m-\beta & 0 \\ 1 & \alpha m & -\beta \end{array} \right) $$
where, I've decided to generalize the Jacobian for all parts of the piecewise linear function by using the relation:
$$ m = \begin{cases} m_1, |y|\geq 1 \\ m_0, |y|\leq 1\\ \end{cases} $$
Is this a right way to go about it?
Q2.The system with values would look like: $$\frac{\mathrm{dx} }{\mathrm{d} t}=-0.2x^2+y^2-z$$ $$\frac{\mathrm{dy} }{\mathrm{d} t}=- h(y)-0.2 y $$ $$\frac{\mathrm{dz} }{\mathrm{d} t}=x+h(y)-0.2 z $$
with equilibrium points given by:
$$x^*=(0;0.3140;-0.3140); y^*=(0;80/49;-80/49); z^*=(0;1.5699;-1.5699);$$
so,in the end how would I choose the distribution of the y* coordinates to find the equilibrium points?(as it is clearly shown,all equilibrium points are functions of y*).
The only thing I can say for certain(as you guys have pointed out) is that y* cannot equal to 1.
Presumably the reason you're computing the Jacobian is to classify equilibria. Recall an equilibrium is a point $(x^*,y^*,z^*)$ at which $\frac{dx}{dt}$, $\frac{dy}{dt}$ and $\frac{dz}{dt}$ are all zero.
If $y^* \ne \pm 1$, then you can evaluate the Jacobian at this equilibrium and compute its eigenvalues and eigenvectors in order to classify the equilibrium and/or determine the local dynamics. Note, the first row of the Jacobian should be $[ -2 a x, 2 y, -\gamma]$, which is slightly different to what you have, but the rest of your expression looks good, including $m$.
Alternatively if $y^* = \pm 1$, then the Jacobian is undefined at the equilibrium. In this case you'll have to do much more work to prove something useful about the local dynamics. However, presumably this case will only arise as a bifurcation.