I am new to convex optimization and have a lot of questions about convex algorithm. Here is what I am confused when reading Prof. Boyd's book (chapter III - Algorithms):
In the book, gradient descent method and steepest descent method are different types. As seen in the figure, the direction of negative gradient and the steepest descent direction are different.
However, what I have known before is that negative gradient points in the steepest descent direction of a level set. Thus, I thought negative gradient $-\nabla f(x)$ must be $-\nabla f(x) = c \Delta x$ where $c$ is a constant. According to the figure as well as the context of the book, $-\nabla f(x)$ does not lies in the same direction of $\Delta x$ in general. Could you please explain that?
A steepest descent direction is always associated to some norm, not necessarily the (standard) Euclidean norm (i.e. $\ell_2$). E.g. steepest descent w.r.t the $\ell_2$ norm is different than steepest descent w.r.t the $\ell_1$ norm (or every other norms).
Keep in mind that:
gradient descent = steepest descent with respect to the $\ell_2$ norm
In the above figure, the steepest descent direction is w.r.t the quadratic norm, which is different than the steepest descent direction w.r.t the $\ell_2$ norm (which is $-\nabla f(x)$).