I have a non-linear cost function $C$ that is a function of a variance-covariance matrix $\Sigma$: $$ C(w) = \frac{\lambda}{2} w^T \Sigma w + \text{...} $$
I'm trying to minimize the cost over the vector $w$ and for the minimization it would be useful to have the Jacobian $J$. What are the partials for this cost function in matrix form? $$ \frac{\partial C}{\partial w_i} $$
Let's use a colon to denote the trace/Frobenius product $$A:B={\rm tr}(A^TB)$$ Write the function in terms of this product and find its differential and gradient $$\eqalign{ C &= \tfrac{\lambda}{2}\Sigma:ww^T \cr dC &= \tfrac{\lambda}{2}\Sigma:(dw\,w^T+w\,dw^T) \cr &= \tfrac{\lambda}{2}(\Sigma+\Sigma^T):dw\,w^T \cr &= \lambda\Sigma:dw\,w^T \cr &= \lambda\Sigma w:dw \cr \frac{\partial C}{\partial w} &= \lambda\Sigma w \cr }$$ The properties of the trace give rise to lots of ways to rearrange the terms in a Frobenius product. For example, all of the following are equivalent $$\eqalign{ A:BC &= AC^T:B \cr &= B^TA:C \cr &= BC:A \cr &= A^T:(BC)^T \cr }$$