Simple interpretation of the direction and projection of the gradient vector

Question

When we use linear approximation of a function, we have a term which includes inner product. For example for the below picture the linear approximation of function $\phi(y)$ at $y=x$ is $\phi(x) + \langle \nabla\phi(x), y - x \rangle$. Since inner product of two vectors is the projection of one vector to the other, by projecting vector $y-x$ on $\nabla\phi(x)$, i.e. the gradient vector at $y=x$, one can find the second term. As it is clear in the picture $$p=\langle \nabla\phi(x), y - x \rangle$$ However, $p$ is not the same as $d$ which is the true value for the increase of function associated to the change from $x$ to $y$. From high school we are sure about $d$. What is wrong?

Answer 1

Note that for a function of one variable the gradient $\nabla \phi(x) = \phi'(x)$ is the derivative that is a scalar quantity not a vector.

The linear approximation is simply

$$L(y)=\phi(x)+\phi'(x)(y-x)$$

Answer 2

You seem to be conflating information provided by the gradient of a function of two variables with information about the derivative of the levelset curve, a function of one variable. The gradient is telling you about the rate of change of the "height" of a surface, one levelset of which is shown in purple. The derivative you have drawn is telling you about the slope of the levelset curve in its plane.

The gradient you have drawn is suitable for a family of levelsets, one of which is shown in purple in your drawing. So the $\nabla \phi$ you have drawn is for a function of the variables $x$ and $y$. In this interpretation, the tangent line you drew is the direction of "no change" in height along the levelset. The gradient is perpendicular to this, indicating that the direction of maximum change is perpendicular to the tangent line you have drawn.

The derivative you have drawn is for $y$ as a function of $x$. This is also the interpretation of your drawing that matches your expression $\phi(y) = \phi(x) + \langle \nabla \phi, y-x\rangle$. Since you only have a single independent variable, this inner product is scalar multiplication, and you have point-slope form for the tangent line: $\phi(y) - \phi(x) = (\nabla \phi)(y-x)$.