Derivative of matrix inverse without inverting matrix

Question

If I have a matrix multiplication $\mathbf{y} = \mathbf{M}\mathbf{x}$, and am given $\mathbf{M}$ and $\mathbf{x}$, can I compute the derivatives $\frac{\partial \mathbf{M}^{-1}\mathbf{y}}{\partial \mathbf{M}_{ij}}$ for all elements of $\mathbf{M}$, without inverting $\mathbf{M}$.

Answer 1

Let $M + te_i e_j^T$ represent a perturbation of $M$. Then, for sufficiently small $t$ we have the series expansion $$(M + te_i e_j^T)^{-1}y = (M (1 + t M^{-1} e_i e_j^T))^{-1}y = \sum_{k = 0}^\infty (-1)^k t^k (M^{-1} e_i e_j^T)^k x \\= x - t M^{-1} e_i e_j^T x + O(t^2)$$ This means that the derivative of $x = (M + t e_i e_j^T)^{-1} y$ with respect to $t$---which is exactly the value you want---equals the $i$th column of $M^{-1}$ scaled by $-e_j^T x$. So, there's no escaping it: if $x$ is non-zero and you know all these partials, then you have computed the inverse of $M$! However, specific partial derivatives can be computed by solving a linear system.