Does newton's method require only the second order partial deriatives of the same variables, or all elements of the Hessian matrix?

65 Views Asked by At

Does Newton's method in optimization require the computation of only the second order partial deriatives of single variables, or all elements of the Hessian matrix?

From Newton's method in optimization, Newton's method performs the iteration $${\displaystyle x_{k+1}=x_{k}+t=x_{k}-{\frac {f'(x_{k})}{f''(x_{k})}}.}$$ So it seems the Newton method's update only requires first and second order gradients. This is the case of XGBoost in Machine Learning:

enter image description here

But some sources I read tell that the Newton method requires Hessian Matrix:

https://www.sciencedirect.com/topics/earth-and-planetary-sciences/hessian-matrix#:~:text=For%20us%20to%20implement%20the,it%20has%20to%20be%20updated

But Hessian Matrix contains second order partial derivatives of two different variables in the non-diagonal elements: $${\displaystyle \mathbf {H} _{f}={\begin{bmatrix}{\dfrac {\partial ^{2}f}{\partial x_{1}^{2}}}&{\dfrac {\partial ^{2}f}{\partial x_{1}\,\partial x_{2}}}&\cdots &{\dfrac {\partial ^{2}f}{\partial x_{1}\,\partial x_{n}}}\\[2.2ex]{\dfrac {\partial ^{2}f}{\partial x_{2}\,\partial x_{1}}}&{\dfrac {\partial ^{2}f}{\partial x_{2}^{2}}}&\cdots &{\dfrac {\partial ^{2}f}{\partial x_{2}\,\partial x_{n}}}\\[2.2ex]\vdots &\vdots &\ddots &\vdots \\[2.2ex]{\dfrac {\partial ^{2}f}{\partial x_{n}\,\partial x_{1}}}&{\dfrac {\partial ^{2}f}{\partial x_{n}\,\partial x_{2}}}&\cdots &{\dfrac {\partial ^{2}f}{\partial x_{n}^{2}}}\end{bmatrix}},}$$

So is the second order partial derivatives of the same variables (or the diagonal elements of Hessian Matrix alone) enough for Newton's method? Are the rest upper/lower triangular elements of Hessian matrix used anywhere in the Newton method as well, e.g., $\dfrac {\partial ^{2}f}{\partial x_{1} \partial x_{2}}$?