What is $\frac{d\|\mathcal{A}(X^TX)-b\|_2^2}{dX}$?

Question

Let $X \in \mathbb{R}^{m\times n}$, $b \in \mathbb{R}^{1\times s}$ and $\mathcal{A}$ a linear operator from $\mathbb{R}^{n \times n}$ to $\mathbb{R}^{1 \times s}$. How do I find $$\frac{d\|\mathcal{A}(X^TX)-b\|_2^2}{dX}?$$ I did a calculation and got $2X\mathcal{A}^*(\mathcal{A}(X^TX)-b)$ but I am not sure.

Answer 1

Let's generalized $b \in \mathbb{R}^{1\times s}$ to $B \in \mathbb{R}^{p\times s}$ and express your linear operator $\mathcal{A}(X'.X)$ as a pre/post matrix multiplication $(G.X'.X.H)$

Also let $M = G.(X'.X).H - B$, so that the function that we wish to differentiate is $$f = \|M\|_F^2 \equiv M:M$$

Take the differential $$ \eqalign { df &= 2M:dM \cr &= 2M:G.d(X'.X).H \cr &= 2G'.M.H':d(X'.X) \cr &= 2G'.M.H':(dX'.X + X'.dX) \cr &= 2G'.M.H':dX'.X + 2G'.M.H':X'.dX \cr &= 2G'.M.H'.X':dX' + 2X.G'.M.H':dX \cr &= 2X.H.M'.G:dX + 2X.G'.M.H':dX \cr &= 2X.(H.M'.G + G'.M.H'):dX \cr }$$ Which means the derivative is $$ \eqalign{ \frac {\partial f} {\partial X} &= 2X.(H.M'.G + G'.M.H') \cr }$$ This could very well be what you meant using the $\mathcal{A^{*}}()$ notation, I'm not familiar with that.