I would like to know in general, and for an example (I will provide) how we can find how many iterations are needed to find the true optimum when using steepest descent on a quadratic given a constant, symmetric matrix with condition number r. Why does this occur?
A specific example;
Let $q(x) = \frac12 x' A x + b'x + c$ where,
$$A = \begin{bmatrix} 1 & 0 & 1 & 1 \\ 0 & 2 & 0 & 0 \\ 1 & 0 & 3 & 0 \\ 1 & 0& 0 & 4 \end{bmatrix}$$
$$b = \begin{bmatrix} 1\\3\\1\\2 \end{bmatrix}$$
The question is actually posed as Use steepest descent method to find the minimiser, starting from $$x = \begin{bmatrix} 2\\2\\2\\2 \end{bmatrix}$$, and to prove that it must be the minimiser.
This isn't something I've encountered in the lectures, but heres my attempt that failed miserably.
I found the condition number to be r = 4.3772/0.34891 = 12.545
Using $q(x_k) - q(x*) <= (\frac{r-1}{r+1})^{2k} [q(x_0) - q(x*)]$
As I want the derivative to be 0, I expect the difference of the kth position (x_k) minus the optimum position (x*) to be 0, so;
$0 <= (0.8523)^{2k} [q(x_0) - q(x*)]$
But this requires that I know the optimum position before finding the optimum position, so obviously will not work.