I have been reading about the hessian for a scholar work about optimization and I find this property:
Let be $H_{P_0}$ the determinant of the hessian matrix for the Lagrangian function $\mathscr{L}(x,y,\lambda)$ evaluated in the critical point $P_0$, then:
- if $H_{P_0}< 0$, $P_0$ in a minimum
- if $H_{P_0}>0 , P_0$ is a maximum
But I didn´t found the prove and I was wondering what happens when the determinat is cero?
In the 1D case, the second derivative test is inconclusive when the second derivative vanishes at the critical point. You can see this by looking at $f(x)=x^3,g(x)=x^4,x_0=0$; the second derivative test gives the same information but the behavior is different. The analogous thing happens in higher dimensions, including in constrained problems.
tl;dr: You can't get any information in this case.