I need to find all of the extreme values of the function
$x^2+y^2+z^2$
constrained to $x^2+2y^2-z^2-1=0$
the problem is that I get this system that I have no idea how to solve:
$2x=2\lambda x$
$2y=4\lambda y$
$2z=-2\lambda z$
$x^2+2y^2-z^2-1=0$
All I can think of is that $x=y=z=0$ but then in the last equation I would get $-1=0$ which of course can't happen.
Any idea what do I need to do here?


There are other candidate solutions, if you select an appropriate value for $\lambda$.
Suppose you solve your first equation for $\lambda$, getting $\lambda=1$. This does not fit with your second or third equation, so you must set $y=z=0$; but you can adjust $x$ to match your final equation and thus get candidates for an extreme point with the zero derivative. You find $x=\pm1, y=z=0$.
You then have the function, with the Lagrnge multiplier built in:
$x^2+y^2+z^2-1(z^2+2y^2-z^2-1)=-y^2+2z^2-1.$
This evidently has a saddle point, not a maximum or minimum, at $(\pm1,0,0)$, so this choice does not work.
But ... you can try a different value of $\lambda$ by solving the second equation first, intending to render $x=z=0$ to fit with that value of $\lambda$. Or you start with the $\lambda$ value obtained from the third equation and put $x=y=0$ to fit with this possible $\lambda$ value.
One of these cases will ultimately lead to a real solution that actually does correspond to an extremum, which you would expect on geometric grounds to be a minimum.