I am wondering how to calculate the gradient of a multivariate Normal respect to L where $\Sigma=LL^{T}$. I know what would be the derivative respect to $\Sigma$ and I can use chain rule to get the derivative respect to L but since $\Sigma$ is a matrix and L is matrix then derivative of a matrix respect to matrix is 4D tensor.
Therefore I am wondering can anyone help to find the derivative of
log|L|$-(x-z)L^{-1}L^{-T}(x-z)^{T}$ respect to L. The first term involving the determinant is a standard calculation but I am stuck in the second term.
In matrix calculus it is often easier to use differentials instead of the chain rule, since it avoids the need to calculate third and fourth order tensors.
Define the variables $$A = LL^T,\quad y = x-z$$ Note that $$A^T=A,\quad A^{-1} = L^{-T}L^{-1} $$ Write the cost function in terms of these new variables. $$\eqalign{ \phi &= \log(\det(L)) + y^TA^{-1}y \\ }$$ Then calculate the differential and the gradient. $$\eqalign{ d\phi &= d\log(\det(L)) + y^TdA^{-1}y \\ &= d{\,\rm tr}(\log(L)) + {\rm tr}\big(y^TdA^{-1}y\big) \\ &= {\rm tr}\Big(d\log(L) - y^TA^{-1}dA\,A^{-1}y\Big) \\ &={\rm tr}\Big(L^{-1}dL-y^TA^{-1}dL\,L^TA^{-1}y-y^TA^{-1}L\,dL^T\,A^{-1}y\Big) \\ &={\rm tr}\Big(L^{-1}dL-2L^TA^{-1}yy^TA^{-1}dL\Big) \\ \frac{\partial \phi}{\partial L} &= L^{-1} - 2L^TA^{-1}yy^TA^{-1} \\ &= L^{-1} - 2L^{-1}yy^TL^{-T}L^{-1} \\ }$$ Depending on your preferred layout convention, you may want the transpose of this result.