Minimizing quadratic form for weighted least squares as a function of weights

Question

In the weighted least squares as a function of weights formulation as:

$$g(w) = \inf_x (Ax -b)^TW(Ax-b) = \inf_x(x^TA^TWAx - 2b^TWAx + b^TWb) $$

where $g(w) $ is the function and $w$ is a set of weights. The weight $ w \in \mathbb R^N $, and $W = \operatorname{diag}(w).$

It can be minimized according to Boyd & Vandenberghe as analytically minimizing the quadratic function to yield: $$ g(w) = b^TWb - b^TWA(A^TWA)^{-1}A^TWb $$

My question is on how to get the last step?

My working thus far is: Minimizing the quadratic form yields: $$ x = (A^TWA)^{-1}b^TWA $$ Substituting this into $g(w)$ does not yield the same simple form?

Answer 1

Actually the solution is $$ x = (A^TWA)^{-1}A^TWb. $$

So that

\begin{align} g = &\; \inf_x(x^TA^TWAx - 2b^TWAx + b^TWb)\\ = &\; b^T W^TA(A^TWA)^{-1}(A^TWA)(A^TWA)^{-1}A^TWb - 2b^TWA(A^TWA)^{-1}A^TWb + b^TWb\\ = &\; b^T W^TA(A^TWA)^{-1}A^TWb - 2b^TWA(A^TWA)^{-1}A^TWb + b^TWb\\ = &\; -b^T W^TA(A^TWA)^{-1}A^TWb + b^TWb. \end{align}

Answer 2

Since the weights should be non-negative, the cost function can be rewritten as follows

$$(\mathrm A \mathrm x - \mathrm b)^\top \mathrm W \, (\mathrm A \mathrm x - \mathrm b) = \| \mathrm W^{\frac 12} (\mathrm A \mathrm x - \mathrm b) \|_2^2 = \| \mathrm W^{\frac 12} \mathrm A \mathrm x - \mathrm W^{\frac 12} \mathrm b \|_2^2 = \| \tilde{\mathrm A} \mathrm x - \tilde{\mathrm b} \|_2^2$$

where $\tilde{\mathrm A} := \mathrm W^{\frac 12} \mathrm A$ and $\tilde{\mathrm b} := \mathrm W^{\frac 12} \mathrm b$. Using the standard normal equations

$$\tilde{\mathrm A}^\top \tilde{\mathrm A} \,\mathrm x = \tilde{\mathrm A}^\top \tilde{\mathrm b}$$

we obtain the normal equations for weighted least-squares

$$\mathrm A^\top \mathrm W \mathrm A \,\mathrm x = \mathrm A^\top \mathrm W \,\mathrm b$$