Minimising a Loss Function requiring Matrix by Matrix Derivative

Question

I am trying to minimise the following cost function with respect to $X$: $\mathbf{C}(X) = ||{M \cdot X \cdot \mathbf{1}_{N \times 1} - T}||_{2}^{2}$

Here, $M$, $X$, $T$ are matrices of dimensions $a \times b$, $b \times n$ and $a \times 1$ respectively. $\mathbf{1}_{N \times 1}$ is a column vector containing all values as $1$.

I have done the following so far: $\begin{align*} \nabla_X \mathbf{C}(X) &= 2 \nabla_X(M \cdot X \cdot \mathbf{1}_{N \times 1} - T) \cdot (M \cdot X \cdot \mathbf{1}_{N \times 1} - T) \\ &= 2 \nabla_X(M \cdot X \cdot \mathbf{1}_{N \times 1}) \cdot (M \cdot X \cdot \mathbf{1}_{N \times 1} - T) \end{align*}$

The idea is to perform a gradient descent once I get the gradient. However, I am not able to evaluate $\nabla_X(M \cdot X \cdot \mathbf{1}_{N \times 1})$. It will be really great if someone can help me with this problem.

Thanks

Answer 1

For convenience, define a new variable $$\eqalign{ Y &= M\cdot X\cdot 1-T \cr dY &= M\cdot dX\cdot 1 \cr }$$ Use this new variable and the Frobenius (:) Inner Product to write the cost function, differential and gradient as $$\eqalign{ C &= Y:Y \cr\cr dC &= 2Y:dY \cr &= 2Y:M\cdot dX\cdot 1 \cr &= 2M^T\cdot Y\cdot 1^T:dX \cr\cr \frac{\partial C}{\partial X} &= 2M^T\cdot Y\cdot 1^T \cr &= 2M^T\cdot (M\cdot X\cdot 1-T)\cdot 1^T \cr }$$

Answer 2

Basically, you require derivative of a vector w.r.t. a matrix. As per wiki, there is no agreement on what the result of differentiation will look like source. Still, here is an ans. Derivative of $AXB$ w.r.t. $X$ is $B^T\otimes A$. Now, you replace $B$ by $I$ and you have your result see second last page .

Edit: It seems that the author of the above linked page missed something to check the matrix conformation.

As mentioned in matrix calculus $d(AXB)=AdXB=(B^T \otimes A)dX$, where $X$ is a vectorized column matrix. As a sanity check, replace $ B=1_{N \times 1}$ in the above. Now, $1_{N \times 1}^T \otimes A_{a \times b}$ has dimension $a \times bN$. $dX$ is a column vector of 1's of size $bN \times 1$. Multiply the two to get your result of consistent dimension.