How can I test the function for three variables if it is convex or concave
Consider $$f(x,y,z)=xz-xyz$$ whete x,y and z $\in (- \infty,\infty) $
A function ƒ is convex or concave if the Hessian matrix of ƒ $H(f)$ is positive definite or negative definite respectively for all values of x, y and z.
then we get the Hessian matrix of ƒ as following,
$H(f)=\left( \begin{array}{ccc} 0 & -z & 1-y \\ -z & 0 & -x \\ 1-y & -x & 0 \\ \end{array} \right)$
since the determinate of $H(f)$ $$D(H(f))=2 x z - 2 x y z$$
since all critical points of $f(x,y,z)$ are $\{(0, 1, z), (0, y, 0), (x, 1, 0)\}$
In this test I can not determine whether the function is positive definite or negative definite.
Is there another test or correction of the method?
where $f:\mathbb{R}^3\rightarrow\mathbb{R} $
Suppose we want to compute the characteristic polynomial of the matrix $H(f)$
$H(f)-\lambda I=\left( \begin{array}{ccc} -\lambda & -z & 1-y \\ -z & -\lambda & -x \\ 1-y & -x & -\lambda \\ \end{array} \right)$
We now compute the determinant of $H(f)-\lambda I$
$$-\lambda ^3+\lambda +\lambda x^2-2 x y z+2 x z+\lambda
y^2-2 \lambda y+\lambda z^2$$

From the 2nd derivative test. The Hessian matrix of $f$ which has negative determinant
Is $f$ neither concave nor convex?

You have already found the hessian of $f$, which is well defined everywhere.
$$H=\begin{bmatrix} 0 & -z & 1-y \\ -z & 0 & -x \\ 1-y & -x & 0 \end{bmatrix}$$
To show that $f$ is convex on a convex domain, it is sufficient to show that $H$ is positive semidefinite on that domain's interior, or, equivalently, that the eigenvalues of $H$ are all nonegative on the interior. We can find the eigenvalues, but this requires a lot of messy algebra, and all we really need is their signs. Because $H$ is symmetric, we can instead use the positive semidefinite case of Sylvester's criterion, which states that $H$ is positive semidefinite iff all the principle minors of $H$ have nonnegative determinants. That is,
$$\det\left({\begin{bmatrix} 0 & -z & 1-y \\ -z & 0 & -x \\ 1-y & -x & 0 \end{bmatrix}}\right)\ge 0$$
$$\det\left({\begin{bmatrix} 0 & -x \\ -x & 0 \end{bmatrix}}\right)\ge 0$$
$$\det\left({\begin{bmatrix} 0 & 1-y \\ 1-y & 0 \end{bmatrix}}\right)\ge 0$$
$$\det\left({\begin{bmatrix} 0 & -z \\ -z & 0 \end{bmatrix}}\right)\ge 0$$
Evaluating determinants:
$$-2x(y-1)z\ge 0$$
$$-x^2\ge 0$$
$$-(y-1)^2\ge 0$$
$$-z^2\ge 0$$
From this, we see that $H$ is positive semidefinite only at the point $(x, y, z)=(0, 1, 0)$. Since this set has no interior, $f$ is not convex anywhere for the typical definition of a convex function of several variables.
To find where $f$ is concave, we determine where $-f$ is convex, which in turn implies $-H$ is positive semidefinite. This only reverses the first inequality, and we obtain the same result, that $f$ is not concave anywhere.