Optimize $f(x,y,z) = xyz + xy$ on $\mathbb{D} = \{ (x,y,z) \in \mathbb{R^3} : x,y,z \geq 0 \wedge x^2 + y^2 + z^2 \leq 1 \}$. The equation $\nabla f(x,y,z) = (0,0,0)$ yields $x = 0, y = 0, z \geq 0 $ and we can evaluate $f(0,0,z) = 0$.
Now studying the function on the boundary $x^2 + y^2 + z^2 = 1$ gets really hairy. I tried replacing $x$ with $\sqrt{1 - y^2 - z^2}$ in order to transform $f(x,y,z)$ into a two-variable function $g(y,z)$ and optimize it on $y^2 + z^2 \leq 1$ but $g(y,z)$ is a pain to differentiate. I then tried spherical coordinates which really did not make it any much easier.
Got any suggestions on how to tackle it?
$f=xy(z+1) \le \dfrac{x^2+y^2}{2}(z+1)=\dfrac{(z+1)(k-z^2)}{2},k\le1$
$f'_z=\dfrac{k-z^2-2z^2-2z}{2}=0,z=\dfrac{-2 \pm \sqrt{4+12k}}{6}=\dfrac{-1 \pm \sqrt{1+3k}}{3}$
for max: $z=\dfrac{-1 + \sqrt{1+3k}}{3},f_{max}=\dfrac{(2+\sqrt{1+3k})(6k-1+2\sqrt{1+3k})}{2*27}\le \dfrac{32}{2*27}$ ,
when $k=1$ get max becasue $6k-1+2\sqrt{1+3k}$ is mono increasing funtion
$z= \dfrac{1}{3}, x=y=\dfrac{2}{3}$