How to start with finding the extremes of the multi-variable function?

Question

Should I start with $G(x,y,z,\lambda)$ ?

function: $f(x, y, z) = xyz$

conditions: $x + y + z = 5$ and $xy + xz + yz = 8$.

Answer 1

I suppose you want to solve with Lagrange? With function $f$ and conditions $g$ and $h$:

$$f_x=\lambda g_x+\mu h_x\\f_y=\lambda g_y+\mu h_y\\f_z=\lambda g_z+\mu h_z\\g(x,y,z)=0\\h(x,y,z)=0$$

This is your system of equations and is often the hardest part. Be careful with the possibility that a variable may be zero.