Derivative involving inverse matrix

Question

Let $\alpha_i\in\mathbb{R}$, $x_i\in\mathbb{R}^d$ for all $i\in[k]$, with $k \geq d$. I am looking for this derivative

$$ \frac{\partial}{\partial\alpha_i} \left(\sum_{i=1}^k\alpha_ix_ix_i^T\right)^{-1} = \frac{\partial}{\partial\alpha_i} \left(X\Lambda X^\top\right)^{-1}, $$

where we define $X: \text{col(X)}= \{x_i\}_{i\in[k]}$, $\Lambda = \text{diag}(\alpha)$, and we assume $\left(X\Lambda X^\top\right)$ is invertible.

Answer 1

Let's call $A=X\Lambda X^T$. Now, $$\frac{\partial}{\partial \alpha_i}A^{-1} = -A^{-1} \left( \frac{\partial}{\partial \alpha_i} A \right)A^{-1}$$ (see Derivative of the inverse of a matrix), provided $A^{-1}$ exists. Finally, $$\frac{\partial}{\partial \alpha_i} A= x_i x_i^T$$.

Answer 2

Given fat matrix ${\rm V} \in \Bbb R^{d \times n}$, let

$${\rm F} ({\rm x}) := {\rm V} \,\mbox{diag} ({\rm x}) {\rm V}^\top, \qquad {\rm G} ({\rm x}) := \left( {\rm F} ({\rm x}) \right)^{-1}$$

Using Sherman-Morrison,

$$\begin{aligned} {\rm G} ({\rm x} + h \, {\rm e}_i) = \left( {\rm F} ({\rm x} + h \, {\rm e}_i) \right)^{-1} = \left( {\rm F} ({\rm x}) + h \, {\rm v}_i {\rm v}_i^\top \right)^{-1} &= {\rm G} ({\rm x}) - h \,\frac{{\rm G} ({\rm x}) \, {\rm v}_i {\rm v}_i^\top {\rm G} ({\rm x})}{1 + h \, {\rm v}_i^\top {\rm F} ({\rm x}) \,{\rm v}_i} \\ &= {\rm G} ({\rm x}) - h \,{\rm G} ({\rm x}) \, {\rm v}_i {\rm v}_i^\top {\rm G} ({\rm x}) + \mathcal O \left( h^2 \right)\end{aligned}$$

and, thus,

$$\partial_i {\rm G} ({\rm x}) = \color{blue}{- {\rm G} ({\rm x}) \, {\rm v}_i {\rm v}_i^\top {\rm G} ({\rm x})}$$

which is what Господин Лисица obtained.

---

matrices matrix-calculus derivatives