Consider the system $$ \dot{x}=x-z+y^2,\quad\dot{y}=x-2y+z+y^2+2x^2z,\quad\dot{z}=-2x+2y+z^2-y^2. $$
and the equilibrium $(0,0,0)$.
Now, there is used some statement that I did not know yet:
Since $$ \frac{\partial (x-z+y^2,x-2y+z+y^2+2x^2z)}{\partial(x,z)}_{|(x,y,z)=0}=\begin{pmatrix}1 & -1\\1 & 1\end{pmatrix}~~~(*) $$ is a non-degenerate matrix, one can find $x$ and $z$ as functions of $y$ from the first two equations: $$ x=y+y^3+O(y^4),\quad z=y-y^2-y^3+O(y^4).~~(**) $$
First of all, what is the left-hand side of $(*)$?
Then, how do we get $(**)$?
It appears that you are not aware of representing derivatives as matrices. This might be helpful. The second question is an exercise in power series approximation.