Derivative wrt to Kronecker Product

Question

Given a function:

$\|\mathbf{y} - (\mathbf{I}\otimes\mathbf{K}) \mathbf{x}\|^2_2$

where $\mathbf{I}$ is an identity matrix, and $\mathbf{K} \in \mathbb{R}^{D \times D}$, how does one find the derivative with respect to $\mathbf{K}$?

Answer 1

First, use the Kronecker-vec operation $${\rm vec}(AMB^T) = (B\otimes A) {\rm vec}(M)$$

to "de-vectorize" the arguments, then find the differential and gradient $$\eqalign{ f &= \|(I\otimes K)x-(I\otimes I)y\|^2_F \cr &= \|KX-Y\|^2_F \cr &= (KX-Y):(KX-Y) \cr\cr df &= 2\,(KX-Y):dK\,X \cr &= 2\,(KX-Y)X^T:dK \cr\cr \frac{\partial f}{\partial K} &= 2\,(KX-Y)X^T \cr\cr }$$ where a colon has been used to denote the Frobenius Inner Product, $x={\rm vec}(X)$ and $y={\rm vec}(Y)$.

Answer 2

Note that $$ \DeclareMathOperator{\tr}{tr} \|y - (I\otimes(K+H)) x\|^2 - \|y - (I\otimes K) x\|^2 =\\ (y - (I\otimes(K+H)) x)^T(y - (I\otimes(K+H)) x) - \|y - (I\otimes K) x\|^2=\\ 2[y - (I \otimes K)x]^T(I \otimes H)x + o(\|H\|) = \\ 2\tr[[(y - (I \otimes K))xx^T]^T(I \otimes H)] + o(\|H\|) $$ So, the Frechet derivative is given by $$ [Df(K)](H) = 2\tr[[(y - (I \otimes K))xx^T]^T(I \otimes H)] $$ How one should write this derivative as a matrix depends on one's choice of notation, but it should be possible to get the desired form from here.