How do we know that the hessian will be positive definite for MLE of logistic regression parameters?

1k Views Asked by At

enter image description here

How does $0<y_n<1$ guarantee that the hessian will be positive definite?

$\Phi^T $ is mxn. then $\Phi^TR\Phi$ is mxm. But This doesn't guarantee positive definite. I suppose since R is a diagonal matrix of all positive values, $\Phi^TR\Phi$ is still positive semi-definite. But what gurantees that $\Phi$ is invertible for us to say that it's positive definite?

1

There are 1 best solutions below

0
On

We see that \begin{align*} v^\intercal\left(\sum_{n=1}^{N} y_n(1-y_n)\phi_n \phi_n^\intercal\right) v = \sum_{n=1}^{N} y_n(1-y_n)(v^\intercal \phi_n)^2 \end{align*} So, since $v^\intercal \phi_n \neq 0$ for at least one $n$ if at least one of $\phi_i, \phi_j$ were not linearly dependent, and $y_n \in (0, 1) \implies y_n(1-y_n) > 0$, the expression can only be 0 whenever $v = \textbf{0}$, proving positive-definiteness.