In [1], I encounter the following statement:
For $ f: \mathbb{R}^{d} \rightarrow \mathbb{R} $ that is twice differentiable,
$ \nabla f (x + p) - \nabla f (x) = \int_{0}^{1} \nabla^{2} f (x + t p) ~ p ~ dt $,
which is named as "Taylor's theorem".
Can someone help me prove the statement?
In particular, does this statement hold even if the function $f$ is more than twice differentiable? I am confused because there could be a nonzero residual term if the function $f$ is more than twice differentiable.
Thanks a lot.
[1] J. Nocedal, S. J. Wright, Numerical Optimization, 2nd Edition, Springer Series in Operations Research and Financial Engineering, Springer, New York, NY, 2006, ISBN-10: 0387303030
On page 70 in [2], we have Theorem 5.6.1 (Taylor's formula with integral remainder). However, this equation is for $f: \mathbb{R} \rightarrow \mathbb{R}$. Thus, let us use the extension in higher dimensions [3] from Wikipedia.
Taylor's Theorem for Multivariate Functions
Let $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$ be $k+1$-times continuously differentiable function in a closed ball $\mathcal{B} = \{\ \mathbf{y} \in \mathbb{R}^{d}: \|\mathbf{x}-\mathbf{y} \| \leq h \}$ for some $h > 0$. Then, for any $\mathbf{x} + \mathbf{h} \in \mathcal{B}$ we have
$ f(\mathbf{x} + \mathbf{h}) = \sum_{|\alpha| \leq k} \frac{D^{\alpha} f(\mathbf{x})}{\alpha!} \mathbf{h}^{\alpha} + \sum_{|\beta| = k + 1} \int_{0}^{1} \frac{(1 - t)^{|\beta| - 1}}{(\beta - 1)!} D^{\beta} f (\mathbf{x} + t \mathbf{h}) \mathbf{h}^{\beta}~dt. $
For the statement
$ \nabla f(\mathbf{x} + \mathbf{h}) - \nabla f (\mathbf{x}) = \int_{0}^{1} \nabla^{2} f (\mathbf{x} + t \mathbf{h}) \mathbf{h}~dt $
in Theorem 2.1 (Taylor's Theorem) of [1], equation (2.5), to hold for any ${f \in \mathcal{C}^{k + 1}}$ with ${\forall k \geq 1}$,
we only need to ensure the notation
$\nabla^{k} f = D^{k} f \in \mathbb{R}^{d_{1} \times ... \times d_{k}}$ hold with ${d_{i} = d}$ for ${i = \{1, ..., k\}}$.
[1] J. Nocedal, S. J. Wright, Numerical Optimization, 2nd Edition, Springer Series in Operations Research and Financial Engineering, Springer, New York, NY, 2006, ISBN-10: 0387303030
[2] Cartan, H. Differential Calculus, Hermann, Paris, 1971.
[3] Königsberger Analysis 2, p. 64 ff. https://en.wikipedia.org/wiki/Taylor%27s_theorem#cite_ref-14