Intuitive understanding of why total derivative gives vector of steepest ascent

Question

I'm reading through http://ufldl.stanford.edu/tutorial/supervised/LinearRegression/

How can I satisfy myself that the highlighted statement is true?

Can anyone Point me towards an intuitive understanding or visualisation?

PS Links:
https://en.wikipedia.org/wiki/Gradient#Differential_or_.28exterior.29_derivative -- doesn't seem to help.
Gradient, tangents, planes, and steepest direction
https://en.wikipedia.org/wiki/Directional_derivative

Answer 1

By the chain rule, the directional derivative of $f$ in the direction $\mathbf{v}$ is $\nabla f\cdot\mathbf{v}=\|\nabla f\|\|\mathbf{v}\|\cos\theta=\|\nabla f\|\cos\theta$ (because a direction vector is a unit vector), where $\theta$ is the angle between the gradient and the direction vector.

This is maximized precisely when $\mathbf{v}$ points in the direction of $\nabla f$ (so that $\cos\theta=1$), and the resulting maximum is

$$\nabla f\cdot\frac{\nabla f}{\|\nabla f\|}=\|\nabla f\|^2/\|\nabla f\|=\|\nabla f\|$$

Answer 2

This is because the directional derivative $D_uJ(\theta)$ in the direction of the unit vector $u$ is equal to $$D_uJ(\theta)=\langle\nabla F(\theta),u\rangle,$$ and the inner product is maximal when the vectors are collinear, with a positive coefficient of proportionality, as is known from the equality case in Cauchy-Schwarz inequality.