Let $f:S\to \mathbb{R}$ be given by $f(x)=x_1x_2x_3 - 2x_1^2-2x_3^2$, with $S:=\{x\in\mathbb{R}^3:-1<x<0\}$. I would like to see if $f$ is a convex function.
One way to determine it is to use the definition of concavity. So let $\theta\in [0,1]$, then
$$f(\theta x + (1-\theta)y)=\theta^3 x_1x_2x_3 - 2\theta^2 x_1^2 - 2\theta^2 x_3^3+(1-\theta)^2 x_1x_2x_3 - 2(1-\theta)^2 x_1^2 - 2(1-\theta^2) x_3^2\ge \theta(x_1x_2x_3-2x_1^2-2x_3^2)+(1-\theta)(y_1y_2y_3-2y_1^2-2y_3^2)=\theta f(x)+(1-\theta)f(y)$$
From which one can see that $f$ is not convex. But not convex does not imply concavity.
Now consider the Hessian matrix of $f$:
\begin{bmatrix} -4 & x_3 & x_2\\ x_3 & 0 & x_1\\ x_2 & x_1 & -4 \end{bmatrix}
Here $D_1=-4, D_2=-x_3^2, D_3=4x_1^2+4x_3^2+2x_1x_2x_3$, so it follows that $D_3\ge 0$, whereas $D_1, D_2\le 0$, so that $H_f$ is indefinite. As far as I understand this implies that $f$ can have a saddle shape.
From both approaches one can say the function is not convex, but what can one conclude about the shape of the graph of $f$? I think we can say that it is not concave using the Hessian approach, or can we?
Would appreciate your insight.
It is not convex because on the path $$x_1 =t , \quad x_2 =x_3 = -\frac{1}{2} , \quad t \in (-1 ,0)$$ is not convex.
It is not concave because on the path $$x_1 =t^2 -1 , \quad x_1 =x_3 = -\frac{1}{2} , \quad t \in (0 ,1)$$ in not concave.