Let's say I'm working with a quadratic function with an equation of $f(\mathbf{x}) = \frac{1}{2}\mathbf{x}^TA\mathbf{x} - b^T\mathbf{x}$. Now, let's take a direction $\mathbf{p}$ and transform the function into one a one dimensional one such that $f'(\alpha) = f(\mathbf{x}_0 + \alpha\mathbf{p)}$, where $\mathbf{x}_0$ is some starting point.
Is there a way to analytically find an $\alpha$ which minimizes or maximizes $f'(\alpha)$? What is instead of arbitrary direction $\mathbf{p}$ we have $\mathbf{p} = -\nabla f(\mathbf{x}_0) $?
(Yes, this is for an optimization problem)
Yes, there is...
$f(\textbf{x})=\frac{1}{2}\textbf{x}^T\textbf{A}\textbf{x}-\textbf{b}^T\textbf{x}$
$\nabla f(\textbf{x})=\textbf{A}\textbf{x}-\textbf{b}$
$g(\alpha)=f(\textbf{x}_0+\alpha\textbf{p})$
$g(\alpha)=\frac{1}{2}\textbf{x}_0^T\textbf{A}\textbf{x}_0+\alpha\textbf{p}^T\textbf{A}\textbf{x}_0+\frac{1}{2}\alpha^2\textbf{p}^T\textbf{A}\textbf{p}-\textbf{b}^T\textbf{x}_0-\alpha\textbf{b}^T\textbf{p}$
Looking for extremum: $g'(\alpha)=0$
$g'(\alpha)=\alpha\textbf{p}^T\textbf{A}\textbf{p}+\textbf{p}^T\textbf{A}\textbf{x}_0-\textbf{b}^T\textbf{p}=0$
$\alpha = -\frac{\textbf{p}^T\textbf{A}\textbf{x}_0-\textbf{b}^T\textbf{p}}{\textbf{p}^T\textbf{A}\textbf{p}}=-\frac{\textbf{p}^T(\textbf{A}\textbf{x}_0-\textbf{b})}{\textbf{p}^T\textbf{A}\textbf{p}}=-\frac{\textbf{p}^T\nabla f(\textbf{x}_0)}{\textbf{p}^T\textbf{A}\textbf{p}}$
And If $\textbf{p}=-\nabla f(\textbf{x}_0)$
$\alpha = \frac{\textbf{p}^T\textbf{p}}{\textbf{p}^T\textbf{A}\textbf{p}}$