Graphically, what is positive semidefinite-ness?

Question

Suppose that we are trying to minimize a function $f$ on $\mathbb{R}^n$ and we apply Newton's method, updating: \begin{align} \mathbf{x}_{n+1} = \mathbf{x}_n - [\nabla^2 f(\mathbf{x}_n)]^{-1} \nabla f(\mathbf{x}_n). \end{align} If the Hessian $\nabla^2 f$ is not positive semidefinite, Newton's method may update $x$ against the gradient because \begin{align} \mathbf{x}_{n+1} - \mathbf{x}_n &= - [\nabla^2 f(\mathbf{x}_n)]^{-1} \nabla f(\mathbf{x}_n) \\ \implies \nabla f(\mathbf{x}_n)^T(\mathbf{x}_{n+1} - \mathbf{x}_n) &= - \nabla f(\mathbf{x}_n)^T[\nabla^2 f(\mathbf{x}_n)]^{-1} \nabla f(\mathbf{x}_n) \end{align} could have either sign. Is there a graphical way to interpret what's going on?

Answer 1

You say you're trying to minimize, but Newton's method actually looks for a critical point (a solution of $\nabla f(x) = 0$). A critical point $x_0$ where the Hessian matrix is not positive semidefinite is not a local minimum but a saddle point or a local maximum: if $v$ is an eigenvector of $\nabla^2 f(x_0)$ for a negative eigenvalue $\lambda$ then $f(x_0 + t v) = f(x_0) + \lambda t^2 |v|^2/2 + O(|t|^3)$ as $t \to 0$.