Help with interpreting the physical meaning of the gradient

Question

I understand that the physical meaning of the gradient is a vector in the direction of maximum increase with the magnitude equivalent to the maximum rate of change. However, I'm simply not seeing it. Is there a good way of visualizing this?

Here is my thinking, which I am visualizing to be a counter example to the interpretation of the gradient. I'm not sure on how to graph this, so I'll explain in words. Assume a function T = x + y, where it is constant in the Z dimension (which allows me to put the T function on the z-axis). At the origin, the function increases equally and the most in both the x direction and the y direction, but along the line of y = x, the function decreases the most.

The gradient of this function at the origin, is the vector addition of the increase in the x direction and increase in the y direction. How then is the vector pointing in $T = <x, y>$ which is along the line of y = x, equivalent to the direction of maximum increase?

Answer 1

Consider two isotimic surfaces $u=c_1$ and $u=c_2$. Now consider the line of shortest distance between these two surfaces. The length of this line is the magnitude and its direction vector from the lower surface to the higher surface gives the direction of the gradient vector.

Answer 2

Here's one way to look at it, if you accept directional derivatives (also has a nice illustration of the gradient).

Suppose we are interested in the direction of greatest increase of some $f(x,y)$. Then, by definition we should look at the rate of increase of $f$ in the direction $v$ via its directional derivative: $$ R_f(v) = \nabla f\cdot \hat{v} $$ which is the rate of change of $f$ in the direction $v$. Intuitively, this is because you are projecting the total change (i.e. the gradient) of $f$ onto the direction vector $v$. It might be obvious that $R_f(v)$ is maximal when $v=\nabla f$, but you can easily see this by letting $$ \hat{v} = (\cos(\theta),\sin(\theta)) = \frac{(v_1,v_2)}{||(v_1,v_2)||_2} = \frac{v}{||v||_2}$$ so that (in 2D) $R_f(v) = \cos(\theta)\partial_xf + \sin(\theta)\partial_yf$. Notice that $\theta$ is simply the angle with the $x$ axis in the plane, so it determines the direction fully. Let's find the maximal $\theta$. Take $\partial_\theta R_f(v) = 0$ and you'll find $\theta=\arctan(\partial_y f/\partial_x f)$, which means $v = (v_1,v_2) = (\partial_x f, \partial_y f)=\nabla f$, as we expected. I.e. $ \max_v R_f(v) = (\nabla f\cdot \nabla f)/||\nabla f||_2 = ||\nabla f||_2 $. So the direction of maximal $R_f$ is $\nabla f$.

---

Ok, well for your specific example, $T(x,y)=x+y$, so $\nabla T = (1,1)$. How can we see that walking along the gradient gives the greatest increase? Let's just take a unit step (from the origin) in a few different directions to see why. Note that $T(0,0)=0$

First, in the $x$ direction, we take a step $s=(1,0)$, so $T(s)=1$. So we took a step of size $1$, and $T$ increased by 1.

Let's try in the $y$ direction: $s=(0,1)$, thus $T(s)=1$, so same as last time.

What about $s=(1,1)/\sqrt{2}$, which is a unit step in the gradient direction? Well, $T(s)=\sqrt{2} > 1$. So even though we took a step of size $1$, our function grew about ${\sim}{1.4}$, rather than just by $1$, as it did in the other directions.

What about something in between? Say, $s=(0.921,0.395)$, so then $T(s)=1.316$. So, taking a step of size $1$ led to an increase of ${\sim}1.3$. This is between the other values, as we would kind of expect.