Derivative of $\operatorname{trace}(XWW^{T}X^{T})$ with respect to $W$

Question

Compute

$$\frac{d}{dW}\operatorname{trace}(XWW^T X^T)$$

where $X$, $W$ are $n\times n$ real matrices.

Answer 1

Let $f : \mathbb R^{n \times n} \to \mathbb R$ be defined by

$$f (\mathrm X) = \mbox{tr} (\mathrm A \mathrm X \mathrm X^T \mathrm A^T)$$

The directional derivative of $f$ in the direction of $\mathrm V$ at $\mathrm X$ is

$$\begin{array}{rl} D_{\mathrm V} f (\mathrm X) &= \mbox{tr} (\mathrm A \mathrm V \mathrm X^T \mathrm A^T) + \mbox{tr} (\mathrm A \mathrm X \mathrm V^T \mathrm A^T)\\ &= \mbox{tr} (\mathrm X^T \mathrm A^T \mathrm A \mathrm V) + \mbox{tr} (\mathrm V^T \mathrm A^T \mathrm A \mathrm X)\\ &= \mbox{tr} ((\mathrm A^T \mathrm A \mathrm X)^T \mathrm V) + \mbox{tr} (\mathrm V^T \mathrm A^T \mathrm A \mathrm X)\\ &= \langle \mathrm A^T \mathrm A \mathrm X, \mathrm V \rangle + \langle \mathrm V, \mathrm A^T \mathrm A \mathrm X \rangle\\ &= \langle 2 \mathrm A^T \mathrm A \mathrm X, \mathrm V \rangle\end{array}$$

Hence,

$$\nabla f (\mathrm X) = 2 \mathrm A^T \mathrm A \mathrm X$$

---

matrix-calculus scalar-fields gradient

Answer 2

A different solution from the one I proposed using the Matrix Cookbook equation $(116)$ (if you are not too familiar with matrix calculus) involves taking these products, then writing them out using index notation:

$$V=WW^T$$ $$A=XV$$ $$B=AX^T$$

Hence: $$v_{ij}=\sum_kw_{ik}w^T_{kj}$$ $$a_{mj}=\sum_ix_{mi}v_{ij}$$ $$b_{mn}=\sum_ja_{mj}x^T_{jn}=\sum_{i,j,k}x_{mi}w_{ik}w_{kj}^Tx_{jn}^T$$

$$\operatorname{trace}(B)=\sum_{m}b_{mm}=\sum_{i,j,k,m}x_{mi}w_{ik}w_{kj}^Tx_{jm}^T$$

Let's suppose we want to find the element of index $(r,s)$ of the resulting derivative matrix. Only two elements of the trace contain the variable $w_{rs}$, namely: $$\sum_{j,m}x_{mr}w_{rs}w_{sj}^Tx_{jm}^T \qquad \text{and}\qquad \sum_{i,m}x_{mi}w_{is}w_{sr}^Tx_{rm}^T$$

All other terms vanish because they are independent of $w_{rs}$. These two sums are in fact the same, hence: $$\frac{d \operatorname{trace}(B)}{w_{rs}}=2\sum_{i,m}x^T_{rm}x_{mi}w_{is}=2\sum_i \left(\sum_m x_{rm}^Tx_{mi}\right)w_{is}$$

The sum over $m$ clearly represents an element of the matrix $X^TX$, so that the sum over $i$ is an element from the matrix $X^TXW$. Finally: $$\frac{d \operatorname{trace}(XWW^TX^T)}{dW}=2X^TXW$$