Say I have a Hessian Matrix:
$\begin{matrix} 0 & 1 \\ 1 & 0 \end{matrix}$
The principal minor for 2x2 symmetric matrix are:
$D_1=0$, $D_2=ac-b^2$
which are: $D_1=0, D_2=-1$.
So, one is negative, one is equal to 0. can I say this function is neither convex and concave ?
If $x_0$ is a point of the domain $D$ of a $C^2$ - function $f$ and if
$H_f(x_0)=\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$, then $H_f(x_0)$ is indefinite.
Thus, $f$ is neither convex nor concave on $D$