Matrix calculus - derivative of scalar function of matrix function of scalar

Question

Cross posted from stats.stackexchange as it wasn't getting any love (given it a few days)!

In the context of REML estimation there is the result (ignoring some constants) that (my interest is in the matrix algebra so some notation is suppressed):

$l(\mathbf V_0)=\log |\mathbf V_0| + \text{tr}(\mathbf V_0^{-1}\mathbf S) \tag{1}$

Where both $\mathbf V_0$ and $\mathbf S$ are symmetric and invertible. I am told that the following expression can be obtained from (1) by differentiating with respect to a parameter $\sigma_i$ of $\mathbf V_0$:

$\text{tr}\left[\mathbf V_0 \frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}\right]-\text{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right] \tag{2}$

I'm trying to see what steps got this, but am stuck.

Now I'm obviously lacking some machinery for dealing with these things but I don't know where to look. The issue is that, say, both terms in (1) are scalar, and we are differentiating w.r.t a scalar, so the it would seem the answer should be scalar. But my workings end up matrix valued. E.g.

$\frac{\partial \log |\mathbf V_0|}{\partial \sigma_i} = \mathbf V_0^{-1} \frac{\partial \mathbf V_0}{\partial \sigma_i} $

Clearly the trace gets involved wrapping it all up into a scalar, but I don't know how or why!

Answer 1

Assuming that $\mathbf S$ is a constant matrix, your formula is wrong. For a counterexample, consider the scalar case $\mathbf V_0=\sigma_i=\sigma$ and $S=1$. We have $l(\mathbf V_0)=\log\sigma+\frac1\sigma$ and $$ \frac{d\,l(\mathbf V_0)}{d\sigma}=\frac1\sigma-\frac1{\sigma^2} =-\left[-\frac1{\sigma^2}(\sigma-1)\right] =-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right]. $$ In other words, your formula has the first summand in excess.

The partial derivative of $l(\mathbf V_0)$ with respect to $\sigma_i$ should be $-\operatorname{tr}\left[\frac{\partial \mathbf V_0^{-1}}{\partial \sigma_i}(\mathbf V_0 - \mathbf S)\right]$. To prove this, recall that the derivative of determinant and derivative of matrix inverse of a matrix function $A$ of a scalar parameter $x$ are given by \begin{align} \frac{d|A|}{dx} &= |A| \operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right),\tag{1}\\ \frac{dA^{-1}}{dx} &= -A^{-1}\frac{dA}{dx}A^{-1}.\tag{2} \end{align} Therefore $$ \frac{d\log|A|}{dx} =\frac1{|A|}\frac{d|A|}{dx} =\operatorname{tr}\left(A^{-1} \frac{dA}{dx}\right) =-\operatorname{tr}\left(\frac{dA^{-1}}{dx}A\right). $$

Answer 2

Rewrite the function into a form more amenable to differentiation $$\eqalign{ f &= \log(\det(V)) + {\rm tr}(V^{-1}S) \cr &= {\rm tr}(\log(V)) + S:V^{-1} \cr }$$ where the colon (:) denotes the Frobenius product.

Letting $d$ denote the derivative operator with respect to $\sigma_i\,\,$ differentiate the above to obtain $$\eqalign{ df &= V^{-1}:dV - S:V^{-1}\,dV\,V^{-1} \cr &= V^{-1}:dV - V^{-1}SV^{-1}:dV \cr &= (V^{-1} - V^{-1}SV^{-1}):dV \cr &= V^{-1}(V-S)V^{-1}:dV \cr &= {\rm tr}\big[V^{-1}(V-S)V^{-1}\,dV\big] \cr }$$