Computing the derivative of a matrix-valued function

Question

Suppose $\mathbf{A} :\mathbb{R}^m\to\mathbb{R}^{m\times m}$ such that $\mathbf{A}(\theta)$ is a symmetric matrix. Let $\mathbf{A}(\theta)$ have the eigen-decomposition $\mathbf{A}(\theta) = \mathbf{U}(\theta)~\mathrm{diag}(\lambda_1(\theta),\ldots,\lambda_m(\theta))~\mathbf{U}(\theta)^\top$. If I have a function $f:\mathbb{R}\to\mathbb{R}$ and I define \begin{align} f(\mathbf{A}(\theta)) = \mathbf{U}(\theta) ~\mathrm{diag}(f(\lambda_1(\theta)),\ldots,f(\lambda_m(\theta)))~\mathbf{U}(\theta)^\top \end{align} How does one compute $\frac{\partial}{\partial\theta_i} f(\mathbf{A}(\theta))$?

Answer 1

For the real function $$F(\theta) \doteq f(A(\theta))$$ the most practical method for calculating the derivative is the Complex Step Approximation $$\eqalign{ \frac{dF}{d\theta} &= {\rm{Im}}\!\left(\frac{F(\theta+i\eta)}{\eta}\right), \qquad \eta &\approx 10^{-90} \\ }$$ unless your function has some special structure which can be exploited.

Update

Invoking the Daleckii-Krein theorem leads to a closed-form expression $$\eqalign{ \def\BR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\l{\lambda} \def\b{\lambda} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} A &= UL\,U^T \qiq L=\Diag{\b_k} \\ F &= f(A) \\ dF &= U\,\BR{R\odot\LR{U^TdA\:U}\!}\,U^T \\ \\ }$$ $$\eqalign{ R_{jk} = \begin{cases} {\Large\frac{f(\b_j)\,-\,f(\b_k)}{\b_j\,-\,\b_k}}\qquad{\rm if}\;\b_j\ne\b_k \\ \\ \quad{f'(\b_k)}\qquad\qquad{\rm otherwise} \\ \end{cases} \\ }$$ where $(\odot)$ denotes the elementwise/Hadamard product and $dA$ can be interpreted as either the differential or as the derivative with respect to $\theta_i$