Gradient of a function as the direction of steepest ascent/descent

Question

I am trying to really understand why the gradient of a function gives the direction of steepest ascent intuitively.

Assuming that the function is differentiable at the point in question,
a) I had a look at a few resources online and also looked at this Why is gradient the direction of steepest ascent? , a popular question on this stackexchange site.
The accepted answer basically says that we multiply the gradient with an arbitrary vector and then say that the product is maximum when the vector points in the same direction as the gradient? This to me really does not answer the question, but it has 31 upvotes so can someone please point out what I am obviously missing?

b) Does the gradient of a function tell us a way to reach the maxima or minima? if yes, then how and which one - maxima or minima or both?
Edit: I read the gradient descent algorithm and that answers this part of my question.

c) Since gradient is a feature of the function at some particular point - am I right in assuming that it can only point to the local maxima or minima?

Answer 1

I had first learned it as if $f(x,y,z) = k$ is a surface $\nabla f$ is a vector perpendicular to the surface.

i.e. the plane tangent to the surface at $\mathbf x = (x_1,y_1,z_1)$ is$\frac {\partial f}{\partial x}(\mathbf x) (x-x_1) + \frac {\partial f}{\partial y}(\mathbf x) (y-y_1) + \frac {\partial f}{\partial z}(\mathbf x)(z - z_1) = 0$

And $(\frac{\partial f}{\partial x}(\mathbf x), \frac{\partial f}{\partial y}(\mathbf x),\frac {\partial f}{\partial z}(\mathbf x))$ is normal to the plane.

$\nabla f$ is a vector perpendicular to the surface when $k$ is fixed. Now we allow $k$ some freedom, and we want to move in the direction of greatest change. Whatever direction we go has a component perpendicular to the surface, and a component parallel to the surface. If we move parallel to the surface we are not contributing to a change in $k.$ The direction of maximal change is $100%$ perpendicular to the surface.

If that intuition isn't working for you. The we are back to the answer you found less than satisfying.

$\frac {\partial f}{\partial x}$ is the change in $f$ for a change in $x.$

For any unit vector $u,$ $\nabla f \cdot u$ would be the change in $f$ for a change in direction $u.$

And we want find $u$ that maximizes $\nabla f \cdot u = \|\nabla f\| cos\theta$

Which will be maximal when $\theta = 0$, or when $u$ points in the same direction as $\nabla f$

Does $\nabla f$ tell us the direction of steepest decent, too? It certainly does. Straight in the opposite direction.

$\nabla f$ does not necessarily point directly toward the local maxima or minima. It points in the direction of greatest change. If you imagine yourself climbing a hill. Straight up the hill is not necessarily the direction the peak of the mountain. You may get up the steep part and then be making a turn.

Answer 2

I think something missing in the accepted answer to the linked question is the connection between gradient and directional derivative. For a function $f(x,y)$ defined at $P=(x_0,y_0)$, and unit a vector $\mathbf{u}=\left<a,b\right>$, we define the directional derivative of $f$ in the direction $\mathbf{u}$ at $P$ to be $$ D_{\mathbf{u}}f(P) = \lim_{t\to 0} \frac{f(x_0 + ta,y_0+tb)-f(x_0,y_0)}{t} $$ I've written it in two variables but the notion carries to any number of variables.

So the question “What is the direction of steepest ascent of $f$ at $P$?” can be translated to: “For which $\mathbf{u}$ is $D_{\mathbf{u}}f(P)$ maximized?”

It turns out there is an easy way to compute $D_{\mathbf{u}}f(P)$ without taking the limit. If you let $g(t) = f(x_0 + ta,y_0 + tb)$, then $D_{\mathbf{u}}f(P)=g'(0)$. And by the Chain Rule, \begin{align*} g'(t) &=\frac{\partial f}{\partial x} \frac{dx}{dt} + \frac{\partial f}{\partial y} \frac{dy}{dt} =\frac{\partial f}{\partial x} a + \frac{\partial f}{\partial y} b\\ \implies g'(0)&= \left<\frac{\partial f}{\partial x}(x_0,y_0),\frac{\partial f}{\partial y}(x_0,y_0)\right>\cdot \left<a,b\right> = \nabla f(x_0,y_0) \cdot \left<a,b\right> \end{align*} To summarize: $$ D_{\mathbf{u}}f(P) = \nabla f(P) \cdot \mathbf{u} $$

What was the question again? That's right: for which $\mathbf{u}$ is $D_{\mathbf{u}}f(P)$ the greatest? Using the identity we just proved, and the law of cosines/dot product formula, we know: $$ D_{\mathbf{u}}f(P) = \left\Vert \nabla f(P) \right\Vert \left\Vert \mathbf{u} \right\Vert \cos\theta = \left\Vert \nabla f(P) \right\Vert \cos\theta $$ where $\theta$ is the measure of the angle between the vectors $\nabla f(P)$ and $\mathbf{u}$. Remember that $\mathbf{u}$ is a unit vector (otherwise $D_{\mathbf{u}}f(P)$ would depend on the length of $\mathbf{u}$, and we really only want it to measure $f$), so $\left\Vert \mathbf{u} \right\Vert=1$.

Now $\cos\theta$ has maximum value $1$, achieved when $\theta = 0$. If the angle between $\nabla f(P)$ and $\mathbf{u}$ has zero measure, the vectors are actually pointing in the same direction. That is: the maximum value of $D_{\mathbf{u}}f(P)$ is $\left\Vert \nabla f(P) \right\Vert$, achieved when $\mathbf{u}$ points in the same direction as $\nabla f(P) $.

Answer 3

This answer is not as detailed as other answers. Start from single dimension. In y=f(x), gradient gives derivative. Now derivative gives direction if steepest ascent. In two dimensions, the partial derivatives are taken to get the direction in both directions individually. So the movement should be in such a direction in which function increases in both dimensions. So you make the vector addition of partial derivatives. Same logic applies to higher dimensions. If you understand that derivative gives direction of steepest ascent then gradient will make sense. Hope this helps!!!

Because derivative gives the direction of tangent at a point. By the definition of derivative, tangent direction is obtained. Now, a tangent is the instantaneous movement direction of a curve. If the movement increases the value of function, keep moving in that direction. Derivative doesn't give Maxima or minima direction. It just gives instantaneous movement direction. Now, if the instantaneous movement gives lesser value of the function, meaning the function is decreasing, go in opposite direction. Meaning negative derivative. In one dimension, it means go in left on x-axis

Answer 4

The question is how you would measure the steepness of ascent. For one-dimensional functions, steepness is defined in terms of the derivative:

$$g^\prime(x) \equiv \lim_{h \rightarrow 0}\frac{f(x+h)-f(x)}{h}$$

By this limit definition, steepness is measured by computing the slope between the points $\langle x, f(x)\rangle$ and $\langle x + h, f(x+h)\rangle$, and letting that distance $h$ get smaller and smaller.

---

Now the question is how we extend this idea of steepness to functions of more than one variable.

Trick #1: Directional steepness requires only ordinary derivatives

Suppose we have a two-variable function $f(x,y)$. (Conceptually, the graph of $f$ is a surface hovering above the $xy$ plane.) Because we are presumably just learning multivariable calculus, we don't have a mathematical definition for the "steepness" at a point $\langle x,y\rangle$. However, there is a trick:

Suppose you pick a point $\langle x_0, y_0\rangle$. And you also pick a direction, in the form of a line like $2y = 3x$. You can see how the height of the function $f$ varies as you start at the point $\langle x_0, y_0 \rangle$ and take small steps in the direction of the line. You can compute this directional steepness using only the ordinary (one-dimensional) derivative.

In fact the equation is something like this:

$$D_{2y=3x} f = \lim_{h\rightarrow 0}\frac{f(x_0 + 2h, y_0 + 3h) - f(x_0, y_0)}{h}$$

(Advanced side note: this definition really is just a one-dimensional derivative. If I parameterize the line $2y=3x$ using a function like $u(t) = \langle 2t, 3t\rangle$, I can define the directional derivative as just $$D_u f \equiv D(f\circ u)(0).$$ To put it in more standard notation, $D_u f \equiv [\frac{d}{dt}f(u(t)) ]_{t=0}$ )

Trick #2: The gradient is a list of the steepness in each axis direction

In the previous section, we defined how to compute the direction steepness of a function — that is, the steepness in the direction of a line.

The lines along the coordinate axes are especially important. If we have a multivariable function $f(x_1, x_2, x_3, \ldots, x_n)$, let $\ell_1, \ell_2, \ldots \ell_n$ be lines, where $\ell_i$ is the line lying along the $x_i$ axis.

We'll define the gradient to be the list of directional steepnesses in each of the coordinate directions:

$$\nabla f = \langle D_{\ell_1}f, D_{\ell_2}f, \ldots, D_{\ell_n}f\rangle.$$

Let's think carefully about this structure. The function $f$ takes in a list of numbers $x_1,\ldots, x_n$ and produces a single number. The function $\nabla f$ takes in a list of $n$ numbers and produces a list of $n$ steepnesses (which are also numbers.)

Visually, you can imagine that $\nabla f$ takes in a point $\langle x_1, \ldots, x_n\rangle$ and produces a steepness vector at that point. The components of that vector are made up of the directional steepnesses of the function $f$ in the direction of the coordinate axes.

Trick #3: Dot products measure directional overlap

When $\vec{u}$ and $\vec{v}$ are vectors, then the dot product between $\vec{u}$ and $\vec{v}$ can be defined by

$$\vec{u}\cdot \vec{v} = ||\vec{u}|| \cdot ||\vec{v} || \cdot \cos{\theta},$$

where $\theta$ is the angle between the two vectors.

Now suppose $\vec{v}$ is kept constant. If we keep the length of $\vec{u}$ constant but allow it to revolve in a circle, for example, we can change the angle $\theta$ and see how it affects the dot product.

Evidently, the dot product is maximized when the two vectors are pointing in the same direction, because then $\cos{\theta}=\cos{0} = 1$ is maximal.

Trick #4: You can compute directional steepness using the dot product

Recall that $D_u f$ is the steepness of $f$ in the direction of some line $u$. Recall that $\nabla f$ is the gradient of $f$— a list of the directional steepnesses in each of the coordinate directions.

It turns out that the following fact is true:

If $u(t) = \langle at, bt\rangle$ is the parametrization of a line, and if $u(t)$ has length 1 when $t=1$, then $$D_u(f) = \nabla f \cdot u(1) $$ In other words, we can compute the directional steepness as the dot product of the gradient and the line of the direction.

Conclusion: The graident is the direction of steepest ascent Because we can compute directional steepness as a dot product with the gradient, the answer to the question: "In which direction is this function steepest?" is the same as the answer to the question "Which line will have the greatest dot product with the gradient?", which we know is "The line which is parallel to the gradient!".

Answer 5

Let’s try coming at it from a different direction, so to speak.

Consider the plane in $\mathbb R^3$ given by $ax+by=z$. The vector $\mathbf n=\langle a,b,-1\rangle$ is normal to this plane. A bit of thought should convince you that the projection of $\mathbf n$ onto the $xy$ plane, $\langle a,b\rangle$, points in the direction in which this plane is steepest. It’s fairly straightforward to prove this analytically, but you can also see this by visualizing cutting a cylinder centered on the $z$-axis with this plane and imagining what happens to the high point of the cut as you tilt the plane in various directions. Displacing the plane from the origin doesn’t change its inclination, so $\langle a,b\rangle$ also gives the steepest direction for any other plane with the same normal, i.e., for $ax+by-z=c$.

Moving now to a curved surface, by analogy with functions of one dimension, we define instantaneous rates of change in terms of tangents to the surface. We’re assuming that the function which defines our surface is suitably well-behaved, so all of these tangents lie in a well-defined tangent plane to the surface. Looking at it another way, this tangent plane captures the rates of change of the function in all directions. As above, then, a “downward” normal to this plane will give us the direction of fastest increase. All we need to do now is find such a normal vector.

Let a surface in $\mathbb R^3$ be given by $F(x,y,z)=c$. Consider a curve $\gamma: t\mapsto(x(t),y(t),z(t))$ on this surface that passes through the point $P_0 = \gamma(0)$, so that we have $(F\circ\gamma)(t)=c$. (Again, we’re assuming that these functions are suitably well-behaved so that this parametrization exists.) Differentiating both sides with respect to $t$ and applying the chain rule gives $$F_x(P_0)x'(0)+F_y(P_0)y'(0)+F_z(P_0)z'(0)=\nabla F(P_0)\cdot\gamma'(0)=0.$$ Now, $\gamma'(0)$ is tangent to $\gamma$ at $P_0$ and thus lies in the tangent plane. Since $\gamma$ was arbitrary, we can conclude that $\nabla F$ is orthogonal to every tangent vector to the surface at $P_0$, i.e., that it is normal to the tangent plane.

For a surface given by $z=f(x,y)$ this normal vector is $\langle f_x,f_y,-1\rangle$, and its projection $\nabla f$ thus points in the direction of steepest ascent along the surface, i.e., the direction in which $f$ increases fastest.

Afterthought: Going back to the original plane example at the top, we can see why this result is plausible. A plane in $\mathbb R^3$ is completely specified by its $x$-slope/rate-of-change $a$, its $y$-slope $b$ and a point on the plane. For the tangent plane to the surface $z=f(x,y)$, these rates of change in the directions of the coordinate axes are given by the partial derivatives of $f$, which are encoded in its gradient.