Let $f(x,y,z)= x^2+y^2+z^2+xyz$.
Show that $f$ is convex on $\Omega=${$(x,y,z)\in R^3 : x^2+y^2+z^2<\frac{5}{2}$}.
To prove it, I want to show that $\nabla^2f(x,y,z)$ is positive definite.
I compute $\nabla^2f(x,y,z)= \left( {\begin{array}{*{20}{c}} 2&z&y \\ z&2&x \\ y&x&2 \end{array}} \right)$.
For $(x,y,z)\in \Omega$, we have $x^2+y^2+z^2<\frac{5}{2}$, so:
$\det(\nabla^2f(x,y,z))=-2(x^2+y^2+z^2)+2xyz+8>-5+2xyz+8=3+2xyz$
I am stuck here.
From the AM-GM inequality, note that $$x^2y^2z^2\leq\left(\frac{x^2+y^2+z^2}{3}\right)^3<\left(\frac{5}{6}\right)^3<1,$$ therefore $$3+2xyz\geq 3-2|xyz|>3-2=1.$$