The derivative and extremum of a matrix function

Question

$$f(W)=(Ax-b)^TW(Ax-b)=x^TA^TWAx-2b^TWAx+b^TWb$$

where $f(W)$ is a function of $W$, $A$ is a known matrix, $x$ and $b$ are vectors ($b$ is known). How to get $\frac{\partial f}{\partial W}$?

Answer 1

Define the vector $$y=Ax-b$$ and write the function in terms of this new variable and the double-dot (aka Frobenius) product.

In this form, the differential & gradient are easy to calculate $$\eqalign{ f &= yy^T:W \cr df &= yy^T:dW \cr \frac{\partial f}{\partial W} &= yy^T \cr }$$

Answer 2

$$f (\mathrm W) := (\mathrm A \mathrm x - \mathrm b)^{\top} \mathrm W (\mathrm A \mathrm x - \mathrm b) = \mbox{tr} \left( (\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top} \mathrm W \right) = \langle (\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top}, \mathrm W \rangle$$

Hence,

$$f ' (\mathrm W) = \color{blue}{(\mathrm A \mathrm x - \mathrm b)(\mathrm A \mathrm x - \mathrm b)^{\top}}$$