Why is the dotproduct of direction and gradient the directional derivative? ($\nabla_\hat{v} f = \nabla f \bullet \hat{v}$)

Question

I have seen the claim that the directional derivative of $f$ in the direction $\hat{v}$, where $||\hat{v}|| = 1$, (denoted $\nabla_\hat{v} f$) is equal to the gradient of $\nabla f$ dotted with $\hat{v}$.

I have tried to prove this to myself, but I got stuck: $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\limit}[2]{\lim_{#1 \to #2}}$ $\newcommand{\pderiv}[2]{\dfrac{\partial#1}{\partial#2}}$

Let $f : \R^n \to \R$

I accept that

$$ \nabla_{\hat{v}} f = \limit{h}{0} \frac{f(x + h\hat{v}) - f(x)}{h}$$

for making intuitive sense. Furthermore I know that

$$ \pderiv{f}{x_i} = \limit{h}{0} \frac{f(x + h\hat{i}) - f(x)}{h} $$

where $\hat{i}$ is the unit vector of the $i$-th dimension. I also know that

$$\nabla f = \left( \pderiv{f}{x_1}, \dots, \pderiv{f}{x_n} \right)$$

Now I want to show that $\nabla_\hat{v} f = \nabla f \bullet \hat{v}$:

\begin{align*} \nabla f \bullet \hat{v} & = \left( \pderiv{f}{x_1}, \dots, \pderiv{f}{x_n} \right) \bullet v\hat{v}\\ &= \sum_{i = 1}^{n} \pderiv{f}{x_i} \cdot \hat{v}_i\\ &= \sum_{i = 1}^{n} \limit{h}{0} \frac{f(x + h\hat{i}) - f(x)}{h} \cdot \hat{v}_i\\ &= \sum_{i = 1}^{n} \limit{h}{0} \frac{\hat{v}_if(x + h\hat{i}) - \hat{v}_if(x)}{h}\\ \end{align*}

And now I am stuck. I don't see a way to transform the last line into $\limit{h}{0} \frac{f(x + h\hat{v}) - f(x)}{h}$ to reach $\nabla_{\hat{v}} f$

Can you help me out here?

---

Edit

I now conceptually understand why the dotproduct of direction and gradient is the directional derivative:

Let's say we have a differentiable function $f$ from $\mathbb{R}^2$ to $\mathbb{R}^3$ mapping the $xy$-plane into the $xyz$-space and we want to know the directional derivative of $f$ at a point $p = (x', y')$ for some vector $u$.

First what we need to know is, how much $f$ changes in $x$ direction and how much it changes in $y$ direction.

Then we need to realize that for small distances whatever direction we go along the surface of $f$, the total change in height is the sum of the change in height in the $x$ component of our direction and the change in height in $y$ component in our direction.

But now we can weight the partial derivatives for the $x$ and $y$ direction with the components of $u$ to get their individual contributions for the direction in which $u$ is pointing!

So if $u$ has an $x$-component of $u_x$ we weight the partial derivative of $f$ for the $x$ direction accordingly. When we do the same for $y$ we get:

$f'(x') \cdot u_x + f'(y') \cdot u_y$

which is in fact $\nabla f \bullet u$

I understood this after listeing to this lecture:

https://www.youtube.com/watch?v=tDPp5uWSIiU

Answer 1

Let $v:=(v_1,v_2,...,v_n)$ and $x:=(x_1,...,x_n)$. Define $y$ with $y_k:=x_k+hv_k$ for $h>0$ and $k=1,...,n$ so that $y=x+hv$. Then $$\frac{df(y_1,...,y_n)}{dh}=\frac{\partial f}{\partial y_1}\frac{dy_1}{dh}+...+\frac{\partial f}{\partial y_n}\frac{dy_n}{dh}=\frac{\partial f}{\partial y_1}v_1+...+\frac{\partial f}{\partial y_n}v_n=\langle\nabla_y f,v\rangle$$ On the other hand $$\frac{df(y_1,...,y_n)}{dh}=\frac{df(x+hv)}{dh}:=g'(h)$$ where $g(h):=f(x+hv)$. Therefore $$g'(0)=\lim_{h\to 0}\frac{f(x+hv)-f(x)}{h}=\frac{df}{dh}\Big|_{h=0}=\langle\nabla_y f,v\rangle\Big|_{h=0}=\langle\nabla_x f,v\rangle$$

Answer 2

There is more in the gradient than the $n$-tuple of partial derivatives!

The function $f:\>{\mathbb R}^n\to{\mathbb R}$ is differentiable at $x$ if there is a linear map $L:\>{\mathbb R}^n\to{\mathbb R}$ such that $$f(x+X)-f(x)=LX\ +o\bigl(|X|\bigr)\qquad(X\to0)\ .\tag{1}$$ This map is then uniquely determined, and is denoted by $df(x)$ (or similar). Since $df(x)$ in this case is a linear functional, by linear algebra there is a vector $a\in{\mathbb R}^n$ such that $$df(x).X= a\cdot X\qquad(X\in{\mathbb R}^n)\ .$$ This vector $a$ is called the gradient of $f$ at $x$, and is denoted by $\nabla f(x)$. Coordinatewise we have $$\nabla f(x)=\left({\partial f\over\partial x_1},\ldots,{\partial f\over\partial x_n}\right)_x\ ,$$ but we shall not need this. Anyway, we now can write $(1)$ in the form $$f(x+X)-f(x)=\nabla f(x)\cdot X\ +o\bigl(|X|\bigr)\qquad(X\to0)\ .\tag{2}$$ Now let a unit vector $e$ be given. Letting $X:=t\,e$ in $(2)$ implies $$f(x+ t e)-f(x)=t\>\nabla f(x)\cdot e +\ o\bigl(|t|\bigr)\qquad(t\to0)\ ,$$ hence $$D_e f(x):=\lim_{t\to0+}{f(x+ t e)-f(x)\over t}=\nabla f(x)\cdot e\ ,$$ as claimed.

Answer 3

$\newcommand{\limit}[2]{\lim_{#1 \to #2}}$ Since this questions does not have an accepted answer yet and I found myself looking for a clear and easily understandable proof with low prerequisites. I found one in 1 which I would like to share with you.

We are trying to prove

$$ \nabla_{\boldsymbol{v}} f(\boldsymbol{x}) = \nabla_{\boldsymbol{x}} f({\boldsymbol{x}}) \cdot \boldsymbol{v}. $$

Define $\boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}}: \mathbb{R} \rightarrow \mathbb{R}^{\mathrm{dim}(\boldsymbol{x})} , \boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}} : h \mapsto \boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}}(h) := \boldsymbol{x} + h \boldsymbol{v}$

and $g_{\boldsymbol{x}, \boldsymbol{v}}: \mathbb{R}\rightarrow \mathbb{R}, g_{\boldsymbol{x}, \boldsymbol{v}} : h \mapsto g_{\boldsymbol{x}, \boldsymbol{v}}(h) := f( \boldsymbol{x} + h \boldsymbol{v} ) = f(\boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}})$

Starting from the definition of the directional derivative,

\begin{align} \nabla_{\boldsymbol{v}} f(\boldsymbol{x}) =& \limit{h}{0} \frac{f(\boldsymbol{x} + h\boldsymbol{v}) - f(\boldsymbol{x})}{h} \\ =& \limit{h}{0} \frac{g_{\boldsymbol{x}, \boldsymbol{v}}(h) - g_{\boldsymbol{x}, \boldsymbol{v}}(0)}{h} \\ =& g_{\boldsymbol{x}, \boldsymbol{v}}'(0) \equiv \frac{\mathrm{d} g_{\boldsymbol{x}, \boldsymbol{v}}} {\mathrm{d} h} \Biggr|_{0} \end{align}

Now consider for an arbitrary $h \in \mathbb{R}$

\begin{align} \frac{\mathrm{d} g_{\boldsymbol{x}, \boldsymbol{v}}(h)} {\mathrm{d} h} =& \frac{\mathrm{d} f\big( \boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}}(h) \big)} {\mathrm{d} h} \\ =& \frac{\mathrm{d} f} {\mathrm{d} \boldsymbol{y_{\boldsymbol{x}, \boldsymbol{v}}}} \cdot \frac{\mathrm{d} \boldsymbol{y}_{\boldsymbol{x}, \boldsymbol{v}}}{\mathrm{d}h} \\ =& \nabla_{\boldsymbol{y}_{\boldsymbol{x}, \boldsymbol{v}}} f \cdot \frac{\mathrm{d} (\boldsymbol{x} + h \boldsymbol{v})}{\mathrm{d}h} \\ =& \nabla_{(\boldsymbol{x} + h \boldsymbol{v})} f \cdot \boldsymbol{v} \end{align}

Which gives for $h=0$

$$ \nabla_{\boldsymbol{v}} f(\boldsymbol{x}) = g_{\boldsymbol{x}, \boldsymbol{v}}'(0) \equiv \frac{\mathrm{d} g_{\boldsymbol{x}, \boldsymbol{v}}} {\mathrm{d} h} \Biggr|_{0} = \nabla_{\boldsymbol{x} } f \cdot \boldsymbol{v} .$$

References:

1 Jorge Nocedal, Stephen J. Wright. Numerical Optimization. Springer, New York, NY. Proof can be found on p. 581 of the second edition. DOI. URL Print ISBN: 978-0-387-98793-4. Online ISBN: 978-0-387-22742-9