The FONC and the optimal step size $\alpha_k$. Help me with the intuition please

Question

I was hoping someone could clarify something for me. For a one-dimensional line search problem

$x^{k+1} = x^{k} + \alpha d_{k}$ where $d_k$ is the descent direction and $\alpha$ the step size, how do we find the optimal step size, $\alpha_k$ at the point $x_k$?

I am confused about a question from my tutorial set and the solution provided by the model answers does not convince me. I argue that just because we find the optimal $\alpha_k$ that does not mean that $x_k+ \alpha_k d_k$ is the point at which we have the global minima so why does the First Order Necessary Condition (FONC) hold? Each step k in the algorithm will have its own optimal $\alpha_k$ and only at the actual optima $x^*$ will the FONC hold

The later steps about orthogonality I follow but I struggle with the intermediate steps

EDIT: Clarified meaning of FONC as the First Order Necessary Condition which means that the directional derivative is zero for a local minima

Answer 1

If I'm not mistaken:

• consider the functions $\phi_k: \mathbb{R} \rightarrow \mathbb{R}$ given by $\phi_k(\alpha)=f(x^k+\alpha d^k)$. Note that we have a different function for every index $k$
• since $\alpha_k$ is a global minimum of the differentiable function $\phi_k$, then $\frac{d}{d\alpha}\phi_k(\alpha_k)=0$. So, we are applying the "FONC" at each step to the composed function$\phi_k$, and that doesn't mean that $x_k+\alpha_k d^k$ is a critical point of $f$ (by coincidence, it would be if your method converged at the current iterate to a local solution of $f$, for instance, but this is generally not the case)