Let $g: \mathbb R^3 \rightarrow \mathbb R$ such that $g(x_1,x_2,x_3)=x_1^2+x_2^2+x_3^2-10x_1x_2x_3$. I know $(0,0,0)$ is critical point and want to check $(0,0,0)$ is local minima or local maxima.
My attempt: I tried from A.M., G.M. inequality but nothing conclusive from it as $\frac{x_1^2+x_2^2+x_3^2}{3}\ge(x_1^2x_2^2x_3^2)^\frac{1}{3}$
I also thought $x_1^2+x_2^2+x_3^2-10x_1x_2x_3 \le ||x||^2-10||x||^3$. I dont know how to proceed further
On
The 3 variable form of the Hessian can be used to check the nature of a critical point
$$H_{xyz} = \begin{bmatrix}\frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x\partial y} & \frac{\partial^2 f}{\partial x\partial z} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} & \frac{\partial^2 f}{\partial y \partial z} \\ \frac{\partial^2 f}{\partial x \partial z} & \frac{\partial^2 f}{\partial y \partial z} & \frac{\partial^2 f}{\partial z^2}\end{bmatrix}$$
Check the determinant value at $(0,0,0)$
Claim:$\;g(x)\ge 0$ for all $x$ with $|x| < {\large{\frac{3}{10}}}$.
Proof:
Assume $|x| < {\large{\frac{3}{10}}}$.
The truth of the claim is immediate if $x_1x_2x_3\le 0$, so assume $x_1x_2x_3 > 0$.
Without loss of generality, we can assume $x_1,x_2,x_3 > 0$.
Let $t=\sqrt[3]{x_1x_2x_3}$.
Then $0 < t\le\sqrt[3]{|x|^3}=|x| < {\large{\frac{3}{10}}}$, so \begin{align*} g(x) &= x_1^2+x_2^2+x_3^2-10x_1x_2x_3 \\[4pt] &\ge 3\sqrt[3]{x_1^2x_2^2x_3^2}-10x_1x_2x_3 \\[4pt] &= 3t^2-10t^3 \\[4pt] &= 10t^2\Bigl({\small{\frac{3}{10}}}-t\Bigr) \\[4pt] &> 0 \\[4pt] \end{align*} which completes the proof of the claim.
It follows that $g$ has a local minimum at $x=(0,0,0)$.