The problem
Let $X$ a submanifold of $\mathbb{R}^3$ defined by $$ X = \left\{ (x,y,z) \mid (x^2 + y^2 + z^2 - 5)^2 = 16(1-z^2) \right\} $$
Let $\Phi(x,y,z) = (x^2 + y^2 + z^2 - 5)^2 - 16(1-z^2) $. Let $f$ a function defined by $$ f : \mathbb{R}^3 \rightarrow \mathbb{R} \mbox{, } (x,y,z) \mapsto x^2 + y^2 + z^2 $$
We define $g$ by the restriction of $f$ on $X$ : $ g = \left. f \right|_X $
Give the points in which the differential of $g$ is zero and deduce the max. and min. points.
My works
$X$ is compact and $g$ is continuous on $X$, then $g$ has a global max. and min. point. Thus, $g$ is $\mathscr{C}^{\infty}$ on $X$. Then, we can apply the method of Lagrange multipliers. We have
$$ \frac{ \partial g }{ \partial x } = 2x \mbox{, } \frac{ \partial g }{ \partial y } = 2y \mbox{, } \frac{ \partial g }{ \partial z } = 2z $$
and
$$ \frac{ \partial \Phi }{ \partial x } = 4x(x^2 + y^2 + z^2 - 5) \mbox{, } \frac{ \partial \Phi }{ \partial y } = 4y(x^2 + y^2 + z^2 - 5) \mbox{, } \frac{ \partial \Phi }{ \partial z } = 4z(x^2 + y^2 + z^2 + 3) $$
We get the following system
$$ \left( \begin{array}{c} 2x\\ 2y\\ 2z\\ \end{array} \right) = \lambda \left( \begin{array}{c} 4x(x^2 + y^2 + z^2 - 5) \\ 4y(x^2 + y^2 + z^2 - 5) \\ 4z(x^2 + y^2 + z^2 + 3) \\ \end{array} \right) $$ with $\lambda$ a real scalar. And I don't know how to resolve this system... Actually, I am not sure if this is the system we have to get. I think I made a mistake somewhere but I don't know where.
Thank you for your help.
Your working is correct -
$\left( \begin{array}{c} 2x\\ 2y\\ 2z \end{array} \right) = \lambda \left( \begin{array}{c} 4x(x^2 + y^2 + z^2 - 5) \\ 4y(x^2 + y^2 + z^2 - 5) \\ 4z(x^2 + y^2 + z^2 + 3) \\ \end{array} \right)$
Also, $(x^2 + y^2 + z^2 - 5)^2 = 16(1-z^2)$.
From above system of equations, please note the possible points to check for max and min.
One is $(x = 0, y = 0, z = 0)$ but it does not meet the fourth equation constraint.
If $x \ne 0$ then from first equation $\lambda = \frac{1}{2(x^2+y^2+z^2-5)}$ but that will lead to $z = 0$ so you have another set of points $(x, y, 0)$.
Similarly if $z \ne 0$ then from the third equation $\lambda = \frac{1}{2(x^2+y^2+z^2+3)}$ but that leads to $x = y = 0$. So the next set of possible points are $(0, 0, z)$.
Taking points $(x, y, 0)$ and plugging into the fourth equation, you have -
$x^2+y^2 - 5 = \pm4 \implies x^2+y^2 = 9$ or $x^2+y^2 = 1$.
Substituting $(0, 0, z)$ in fourth equation shows no real solution.
So max and min of $x^2 + y^2 + z^2$ for the given equality constraint seem to be $9$ and $1$ resp, and for all points on $x^2 + y^2 = 9, z = 0$ and $x^2 + y^2 = 1, z = 0$ resp.