Prove $f(x^{k+1}) < f(x^k)$ in gradient descent method.

Question

Let $f(x)$ be represented with a definite positive symmetric square matrix $Q$. The gradient descent method for finding the minimum of $f(x)$ ensures that $f(x^{k+1}) < f(x^k)$. Prove that $f(x^{k+1}) < f(x^k)$ is true using $f(x^{k+1}) = f(x^k) - \frac{1}{2} \frac{||\nabla f(x^k)||^4 }{||\nabla f(x^k)||^2_Q }$.

Answer 1

I'm not entirely sure what you mean by that $Q$ as an index to the norm.

But if we are looking at norms of the gradient of $f$ then we have

$$\frac{||\nabla f(x^k)||^4}{||\nabla f(x^k)||_Q^2} > 0 $$

So you are substracting something positive from $f(x^k)$ to get $f(x^{k+1})$, hence $f(x^{k+1})$ must be smaller than $f(x^k)$.