Derivative of a function with respect to a matrix

Question

I need to calculate derivative of the following function with respect to the matrix X:

$f(X)=||diag(X^TX)||_2^2$

where $diag()$ returns diagonal elements of a matrix into a vector. How can I calculate $\frac {\partial f(X)} {\partial X}$? Please help me.

Thanks in advance!

Answer 1

In terms of the Hadamard ($\circ$) and Frobenius ($:$) products, the function can be written as $$ \eqalign { f &= \|I\circ(X^TX)\|^2_F \cr &= I\circ(X^TX):I\circ(X^TX) \cr &= I\circ(X^TX)\circ I:(X^TX) \cr &= I\circ I\circ(X^TX):(X^TX) \cr &= I\circ(X^TX):(X^TX) \cr &= I:(X^TX)\circ(X^TX) \cr }$$ The differential of which is $$ \eqalign { df &= I:2\,(X^TX)\circ d(X^TX) \cr &= 2\,I\circ(X^TX):d(X^TX) \cr &= 2\,I\circ(X^TX):(dX^TX+X^TdX) \cr &= 2\,I\circ(X^TX):2\,{\rm sym}(X^TdX) \cr &= 4\,{\rm sym}(I\circ(X^TX)):X^TdX \cr &= 4\,X(I\circ(X^TX)):dX \cr }$$ Since $df = \big(\frac {\partial f} {\partial X}\big):dX\,\,$ the derivative must be $$ \eqalign { \frac {\partial f} {\partial X} &= 4\,X(I\circ(X^TX)) \cr }$$ It helps to know a few simple rules for manipulating these products.

Unlike the regular matrix product, both the Hadamard and Frobenius products are commutative $$ \eqalign { A\circ B &= B\circ A \cr A:B &= B:A \cr }$$ They also satisfy these mixed-product rules $$ \eqalign { A:BC &= AC^T:B \cr A:BC &= B^TA:C \cr A\circ B:C &= A:B\circ C \cr }$$ The latter being sort of an element-wise triple (scalar) product.

Answer 2

Here you get a scalar function of the matrix $X$. The derivative of such a scalar function is defined as the matrix of partial derivatives wrt each entry of the matrix, as here.

You can expand out the expression and calculate the partial derivative.