Im trying to optimize
$f(x,y,z)=xyz$ restricted to $g(x,y,z)=x+y+z=1$ and $h(x,y,z)=x^2+y^2+z^2=1$.
$∇f=(yz,xz,xy)$, $∇g=(1,1,1)$ and $∇h=(2x,2y,2z)$.
I tried using the determinant $det(∇f,∇g,∇h)=yz(2z-2y)-xz(2z-2x)+xy(2y-2x)=0$ which I dont know what to do with and I cant simplify the determinant in a good way with row operations.
I also tried solving $z$ from $g=1$. $z=1-x-y$.
$f(x,y,1-x-y)=xy-x^2y-xy^2$ restricted to $h(x,y)=2x^2+2y^2-2x-2y+2xy+1$ with Lagrange multiplier but I made no progress there either as the partial derivatives got too messy.
The Lagrange multiplier method is effective for this problem. First of all observe that the case $x=y=z$ is inadmissible. By symmetry we may assume that $z\neq x$ and $z\neq y.$ We have to solve for $$\nabla(xyz)=\lambda \nabla(x+y+z)+\mu\,\nabla(x^2+y^2+z^2) $$ i.e. \begin{eqnarray*} yz &=& \lambda +2\mu x\\ xz &=& \lambda +2\mu y \\ xy &=& \lambda +2\mu z \end{eqnarray*} Subtracting the third equation from the first two ones gives \begin{eqnarray*}y(z-x)=-2\mu(z-x)\\ x(z-y)=-2\mu(z-y) \end{eqnarray*} Therefore $x=y.$ We thus get $$2x+z=1,\quad 2x^2+z^2=1$$ This leads to two solutions $x=y=0,$ $z=1$ and $x=y={2\over 3},$ $z=-{1\over 3}.$ Thus $$M=0,\qquad m={2\over 3}\cdot {2\over 3}\cdot \left (-{1\over 3}\right )=-{4\over 27}$$