Minimizing a quadratic form under linear constraints
Let $\textbf{A}\in \mathbb{R}^{n\times n}$ be symmetrical positive definite. Let us minimize $$ f(\textbf{x}) := 2^{-1}\textbf{x}^\top\textbf{A}\textbf{x} $$ over all $x \in \mathbb{R}$ fulfilling the condition $\textbf{Bx} = \textbf{c}$, where $\textbf{B}$ is a matrix in $\mathbb{R}^{q\times n}$ with rank $q\leq n$, and $\textbf{c}$ is a vector in $\mathbb{R}^q$. For $\pmb{\lambda} \in \mathbb{R}^q$, let us minimize the function $L(\textbf{x}, \pmb{\lambda}) = f(\textbf{x}) + \pmb{\lambda}^\top \textbf{Bx}$ expanded below: \begin{align*} L(\textbf{x}, \pmb{\lambda}) &= 2^{-1}\textbf{x}^{\top}\textbf{Ax} + (\textbf{B}^\top \pmb{\lambda})^\top\textbf{x} \quad \star 1\\ &= 2^{-1}\big(\textbf{x}^{\top}\textbf{Ax} + 2(\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}) \textbf{A}\textbf{x}\big) \quad \star 2\\ &= 2^{-1}(\textbf{x}+ \textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda})^\top\textbf{A}(\textbf{x}+ \textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}) - 2^{-1}\pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda} \quad \star 3 \end{align*} $L(\cdot, \pmb{\lambda})$ is minimized exactly when $\textbf{x}$ equals $\textbf{x}_{\lambda} := -\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}$. Furthermore, $$ \textbf{Bx}_\lambda= -\textbf{BA}^{-1}\textbf{B}^\top\pmb{\lambda} $$ and equals $\textbf{c}$ if and only if $\pmb{\lambda} = -(\textbf{B}\textbf{A}^{-1}\textbf{B}^\top)^{-1}\textbf{c}$. As a consequence, the problem of origin has the unique solution $$ \textbf{x}^\star := \textbf{A}^{-1}\textbf{B}^\top(\textbf{B}\textbf{A}^{-1}\textbf{B}^\top)^{-1}\textbf{c} \quad \text{with} \quad f(x^\star) = 2^{-1}\textbf{c}(\textbf{B}\textbf{A}^{-1}\textbf{B}^\top)^{-1}\textbf{c} $$
Here is what I don't understand:
- How can you go from $\star 1$ to $\star 2$?
- How can you go from $\star 2$ to $\star 3$?
There's a typo in $\star2$. It should read $$ 2^{-1}\big(\textbf{x}^{\top}\textbf{Ax} + 2(\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda})^\top \textbf{A}\textbf{x}\big)\ . $$ But, in any case, if you want to show that $\ \star1\ $ and $\ \star3\ $ are equal, getting there by way of $\ \star2\ $ (even as amended) seems to me to be an unnecessarily roundabout way of going about it. A much simpler way of doing it is just to expand out the expression for $\ \star3\ $ and appeal to some well-known matrix identities.
Using the fact that $\ \left(\textbf{C}\textbf{D}\right)^\top\ = \textbf{D}^\top\textbf{C}^\top\ $ and $\ \left(\textbf{C}^\top\right)^\top = \textbf{C}\ $for any matrices $\ \textbf{C}\ $ and $\ \textbf{D}\ $ for which the product is well-defined, we get \begin{eqnarray} \left(\textbf{B}^\top\pmb{\lambda}\right)^\top &=& \pmb{\lambda}^\top\textbf{B}\mbox{ , and}\\ \left(\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}\right)^\top&=& \pmb{\lambda}^\top\textbf{B}\left(\textbf{A}^{-1}\right)^\top\\ &=& \pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1}\ , \end{eqnarray} because the symmetry of $\ \textbf{A}\ $ implies that $\ \textbf{A}^{-1}\ $ is also symmetric. And because $\ \pmb{\lambda}^\top\textbf{B}\textbf{x}\ $ is a scalar, we have \begin{eqnarray} \pmb{\lambda}^\top\textbf{B}\textbf{x} &=& \left(\pmb{\lambda}^\top\textbf{B}\textbf{x}\right)^\top\\ &=& \textbf{x}^\top\textbf{B}^\top\pmb{\lambda} \end{eqnarray} So, \begin{eqnarray} \star 3 \quad &=&2^{-1}(\textbf{x}+ \textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda})^\top\textbf{A}(\textbf{x}+ \textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}) - 2^{-1}\pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}\\ &=& 2^{-1}\big(\textbf{x}^{\top}\textbf{Ax} + \pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1} \textbf{A}\textbf{x} + \textbf{x}^{\top}\textbf{A}\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}\\ & &\ \ \ \ \ \ \ \ \ \ +\pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1}\textbf{A}\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}\big) - 2^{-1}\pmb{\lambda}^\top\textbf{B}\textbf{A}^{-1}\textbf{B}^\top\pmb{\lambda}\\ &=& 2^{-1}\big(\textbf{x}^{\top}\textbf{Ax} + \pmb{\lambda}^\top\textbf{B}\,\textbf{x} + \textbf{x}^\top\textbf{B}^\top\pmb{\lambda}\big)\\ &=& 2^{-1}\textbf{x}^{\top}\textbf{Ax} + \left(\textbf{B}^\top\pmb{\lambda}\right)^\top\textbf{x}\quad\star 1\ , \end{eqnarray} by appealing to the identities listed above.