I have an expression that I need to find the derivative of, but I can't work out how to derive it as the variable I am wanting the gradient with respect to is inside a vec operator. Such that:
$tr(\mathbf{A}vec(\mathbf{S}\mathbf{X})vec(\mathbf{S}\mathbf{X})^T\mathbf{A}^T)$
$\mathbf{A} \text{ and } \mathbf{X}$ are known matrices, and I want the derivative of this expression with respect to $\mathbf{S}$.
How does one go about this?
Let $$\eqalign{ v &= {\rm vec}(SX) = {\rm vec}(ISX) \cr &= (X^T\otimes I)\,{\rm vec}(S) = (X^T\otimes I)\,\sigma \cr\cr }$$ Write the function in terms of the Frobenius Product, then find its differential and gradient $$\eqalign{ f &= Av:Av \cr\cr df &= 2\,Av:Adv \cr &= 2\,A^TAv:dv \cr &= 2\,A^TAv:(X^T\otimes I)\,d\sigma \cr &= 2\,(X\otimes I)\,A^TAv:d\sigma \cr\cr \frac{\partial f}{\partial\sigma} &= 2\,(X\otimes I)\,A^TAv \cr\cr }$$ To recover the gradient wrt the matrix $S$, "matricize" the vector result by unstacking the columns $$\eqalign{ \frac{\partial f}{\partial S} &= {\rm Mat}\Big(\frac{\partial f}{\partial\sigma}\Big) \cr }$$ where $${\rm Mat}\big(\sigma\big)={\rm Mat}\big({\rm vec}(S)\big)=S$$