Can the step in gradient descent be negative?

605 Views Asked by At

For a single-variable function, gradient descent works just by repeatedly computing the next x that is closer and closer to the minimum of the function.The formula is

$$x_{i+1}:=x_i-t\nabla f\vert_{x= x_i} $$

But can "t" be negative or is it just positive? Does the gradient lead the way to the minimum so t is just a number to specify the magnitude of the step?

Thanks!

1

There are 1 best solutions below

0
On BEST ANSWER

Your intuition is correct: $t$ specifies the magnitude of the step. If you make the step size negative, you're now walking backwards, away from the minimum. This is equivalent to gradient descent of the function $-f$.

There are cases when it is useful to vary step size. When step size is similar to the distance to the minimum, $x_i$ will "zig-zag" around the minimum without settling on it. So you might begin a gradient descent algorithm with a large step size to find the general neighborhood of a minimum, then gradually decrease the step size to refine your estimate.