Gradient of matrix product with respect to matrix

Question

If a matrix $Y$ is defined as the product of two matrices $X$ and $W$ ie $Y=X.W$ , how do we compute the gradients $\partial Y/\partial W$ and $\partial Y/\partial X$. The dimensions of matrix $X$ is $m*d$,those of $W$ are $d*h$ and those of $Y$ $m*h$ .

I have a basic knowledge of matrices and their multiplication but have no knowledge of matrix calculus. Any recommendation about books regarding this subject would be appreciated. My current field of study is neural networks and I need to study matrix calculation to perform backpropagation.

Answer 1

I would write it this way: $F(X, W) = X W$, $$DF(X, W)\cdot(H, K) = H W + X K$$ where $DF$ is the differential of $F$. This notation is less ambiguous than partial derivatives. Note that for a continuous bilinear map, one alway has $$DF(X, W)\cdot (H, K) = F(H, W) + F(X, K)$$ One can also use $dF$ or $F^\prime$ instead of $DF$. The partial derivative $\partial F/\partial X$ would be the linear map $H\mapsto H W$.

Answer 2

The gradient of a matrix with respect to a matrix is a 4th order tensor, so it's best to use index notation. $$\eqalign{ Y_{ik} &= X_{ij}W_{jk} \cr dY_{ik} &= dX_{ij}\,W_{jk} + X_{ij}\,dW_{jk} \cr \cr \bigg(\frac{\partial Y}{\partial X}\bigg)_{ikpq} &=\frac{\partial Y_{ik}}{\partial X_{pq}} = \frac{\partial X_{ij}}{\partial X_{pq}}\,W_{jk} = \delta_{ip}\,\delta_{jq}\,W_{jk} = \delta_{ip}\,W_{qk} \cr \cr \bigg(\frac{\partial Y}{\partial W}\bigg)_{ikpq} &= \frac{\partial Y_{ik}}{\partial W_{pq}} = X_{ij}\,\frac{\partial W_{jk}}{\partial W_{pq}} = X_{ij}\,\delta_{jp}\,\delta_{kq} = X_{ip}\,\delta_{qk} \cr \cr }$$