Derivative of trace involving Hadamard product

199 Views Asked by At

Let us assume that $A, S\in\mathbb{R}^{n\times n}$, $U\in\mathbb{R}^{n\times k}$, and $V\in\mathbb{R}^{n\times k}$. I am trying to differentiate the following expression: $$\Phi(U,V)=\mathrm{trace}\left((S\circ A)(S^T\circ(VU^T))\right),$$ with respect to $U$ and $V$, respectively, i.e.: $$\frac{\partial \Phi}{\partial U}\quad and\quad \frac{\partial \Phi}{\partial V}.$$

in which $\mathrm{trace}(\cdot)$ is the trace of a matrix, and $\circ$ is the Hadamard product.

I appreciate any help.

1

There are 1 best solutions below

1
On BEST ANSWER

$\def\T{\operatorname{Tr}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$For typing convenience, let's define the matrix variable $$B=S\circ S\circ A$$ and use a colon to denote the trace/Frobenius product $$\eqalign{ A:B &= \T(A^TB) \\ A:A &= \big\|A\big\|_F^2 \\ }$$ The properties of the trace allow the terms in such a product to be rearranged $$\eqalign{ A:B &= B:A = B^T:A^T \\ CA:B &= C:BA^T = A:C^TB \\ }$$ Further, the Frobenius and Hadamard products commute $$A:(B\circ C) = (A\circ B):C$$ Write the cost function using the above notation, then calculate its differential and gradients. $$\eqalign{ \phi &= (S\circ A):\big(S\circ UV^T\big) \\&= B:UV^T \\ d\phi &= B:\big(dU\,V^T+U\,dV^T\big) \\&= BV:dU \;+\; B^TU:dV \\ \p{\phi}{U} &= BV \qquad\quad \p{\phi}{V} = B^TU \\ }$$