Differentiate square matrix $VV^T \in R^{n \times n} $ over rectangular matrix $V \in R^{n \times r}$

Question

I want to differentiate $f = logdet(L)$ by $V$ where $L = VV^T$.

The thing that I know is $\frac{df}{dv_{ik}} = tr(L^{-1} \frac{dL}{dv_{ik}})$, where ${\{\frac{df}{dV}\}}_{ik} = \frac{df}{dv_{ik}}$.

The problem is the upper one in $\frac{dL}{dv_{ik}}$ is a square matrix but the down one is a rectangular matrix. How can i derive the solution?

Thank you in advance.

Answer 1

In matrix calculus, the chain rule is not your friend. The main problem is that the gradient of a matrix with respect to another matrix is an awkward, higher order tensor.

It's much easier to work with differentials, since the differential of a matrix behaves like an ordinary matrix.

Let's start with the differential of $L$ $$\eqalign{ L &= VV^T \cr dL &= dV\,V^T + V\,dV^T \cr }$$ We can use this to find the differential of your function $$\eqalign{ f &= \log\det L \cr df &= d\log\det L = d\,{\rm tr}\log L \cr &= L^{-T}:dL \cr &= L^{-T}:(dV\,V^T + V\,dV^T) \cr &= (L^{-1}+L^{-T}):dV\,V^T \cr &= 2L^{-1}V:dV \cr \frac{\partial f}{\partial V} &= 2L^{-1}V = 2(VV^T)^{-1}V = 2(V^T)^+ \cr }$$ In the preceding, $A^+$ denotes the Moore-Penrose inverse of $A$, and a colon is used to denote the trace/Frobenius product, i.e. $\,\,A:BC={\rm tr}(A^TBC)$.

The properties of the trace give rise to many rules for rearranging the arguments of a Frobenius product, such as $$\eqalign{ A:BC &= B^TA:C = AC^T:B \cr &= A^T:(BC)^T = BC:A \cr }$$

Answer 2

Here $n<r$ and $rank(V)=n$. One has

$Df_V:H\in M_{n,r}\rightarrow Df_V(H)=tr((VV^T)^{-1}(HV^T+VH^T))$ and

$\dfrac{\partial f}{\partial V_{i,k}}=tr((VV^T)^{-1}(E_{i,k}V^T+VE_{k,i}))$.