How Is the Jacobian squared ( truncated Hessian) approximately equal to the Hessian

Question

I'm currently work on second order optimization. And I'm stuck at quass Newton method. I'm are trying to approximate the Hessian without actually computing the Hessian . Please tell how can the Hessian be represented using the gradient or Jacobian

Answer 1

Nonlinear least squares solves a problem of the form minimize $\frac{1}{2} \Sigma_{i=1}^m R_i(x_1,...,x_n)^2$

The Jacobian, $J$, of the objective function being minimized is the matrix whose (i,j) element = $\frac{\partial{R_i(x_1,...,x_n)}}{\partial{x_j}}$

The Hessian, $H$, of the objective function being minimized = $J^TJ + \Sigma_{i=1}^m R_i(x_1,...,x_n) * \text{Hessian}(R_i(x_1,...,x_n)).$. The 2nd term consists of "2nd order" terms, and is of small magnitude for $x$ near the optimum solution of the problem, provided the residuals, $R_i(x_1,...,x_n)$, are small in magnitude at the solution (i.e., low residual problem). Under these circumstances, the Hessian can be approximated by $J^TJ$, and it usually works fairly well, unless the residuals are too large. Such approximation to the Hessian is used in the Gauss-Newton and Levenberg-Marquardt algorithms.

Quasi-Newton algorithms build up an approximation to the Hessian (or in some cases, the Hessian inverse) of a general (twice continuously differentiable) objective function, by using the differences of gradients across several optimization algorithm iterations. Different Quasi-Newton algorithms do this in different ways. The Quasi-Newton Hessian approximation may perform well in its role in the optimization algorithm, but is not necessarily a close approximation to the true Hessian.