I want some clarification on the second order Taylor theorem for scalar fields. Everywhere I searched for, either uses the remainder form, or an "approximation sign" (which I find very imprecise, to be honest). So, I want to check if the following result that I state below, is correct or not:
Theorem: Let $f: \mathbb{R}^d \mapsto \mathbb{R}$ be a function, which has derivatives up to the second order (you can assume continuity of second derivatives, or even existence of third derivatives, if needed; it does not matter to me, since my function is infinitely differentiable). Then, for any two vectors $x,y \in \mathbb{R}^d$, there exists a vector $v$ such that $\|v-x\|_1 < \|y-x\|_1$, and $$f(y) - f(x) - \nabla f(x)^\top (y-x) = \frac{1}{2}(y-x)^\top \nabla^2 f(v) (y-x)~,$$ where $\nabla f$ and $\nabla^2 f$ denote the gradient vector and the Hessian matrix of $f$, respectively.
My main concern is the correctness of the part $\|v-x\|_1 < \|y-x\|_1$. This would be guaranteed, if for instant, one had that the vector $v$ lies on the straight line segment in $\mathbb{R}^d$ joining $x$ and $y$ (is this last part true?) Note that, for a vector $v = (v_1,\ldots,v_d)$, $$\|v\|_1 := \sum_{i=1}^d |v_i|~.$$ Can you please tell me if the above theorem is correct, and if so, give a proper reference? Thanks in advance.
Yes, it's true, and one can choose $v$ to lie on the line segment joining $x$ and $y$. For reference, take a look at Loomis and Sternberg's Advanced Calculus, Section $3.17$, bottom of page $192$ to page $193$.
There, the theorem is expressed in slightly different notation, but the proof is simple, so I'll repeat the relevant bits. Fix $x,y \in \Bbb{R}^n$, and define the function $\gamma:\Bbb{R} \to \Bbb{R}^n$ by $\gamma(t) = x + t(y-x)$. If $t \in [0,1]$, then $\gamma$ parametrizes the line segment going from $x$ to $y$. Now, consider the composite function $f \circ \gamma : \Bbb{R} \to \Bbb{R}$. The idea is to apply Taylor's theorem to this function: \begin{align} (f \circ \gamma)(1) &= (f \circ \gamma)(0) + (f \circ \gamma)'(0)\cdot (1-0) + \dfrac{1}{2!} (f \circ \gamma)''(\tau) \cdot (1-0)^2, \end{align} for some $\tau \in [0,1]$. This is simply by the standard single variable version of Taylor's remainder formula (I forget if this is called the Cauchy/Lagrange remainder).
If you calculate these derivatives explicitly, you'll see that this reduces to \begin{align} f(y) &= f(x) + (\nabla f)_{\gamma(0)} \cdot (y-x) + \dfrac{1}{2} (y-x)^T\cdot (\nabla^2f)_{\gamma(\tau)} \cdot (y-x) \\ &= f(x) + (\nabla f)_x \cdot (y-x) +\dfrac{1}{2} (y-x)^T\cdot (\nabla^2f)_{x + \tau(y-x)} \cdot (y-x), \end{align} where the subscript refers to the point of evaluation of the derivatives. So, we choose $v= x + \tau(y-x)$, for some $\tau \in [0,1]$.