What is the second derivative of the spectral norm of a symmetric matrix?

Question

It is well known that the derivative of a matrix $A$'s $2$-norm with respect to $A$ is $$\frac{\partial \sigma_{\max}(A)}{\partial A}=u_1v_1^T,$$ where $u_1,v_1$ are the first column/row in the SVD matrices. However, I want to know if we have an explicit formula for the second derivative of $\sigma_{\max}(A)$?

Answer 1

$ \def\R#1{{\mathbb R}^{#1}} \def\o{{\tt1}} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\rank#1{\op{rank}\LR{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\hess#1#2#3{\grad{^2 #1}{#2\,\p #3}} \def\Hess#1#2{\grad{^2 #1}{#2^2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} \def\fracLR#1#2{\LR{\frac{#1}{#2}}} \def\gradLR#1#2{\LR{\grad{#1}{#2}}} \def\Sk{\sum_{k=\o}^{\rank A}} \def\s{\sigma} \def\k{\otimes} \def\Im{I_m} \def\In{I_n} $Summarizing the results from this post $$\eqalign{ A &= \Sk \s_k\, u_k\, v_k^T,\quad &A \in\R{m\times n},\;\;u_k\in\R m,\;\;v_k\in\R n \\ a &= \vc A,\qquad &\grad{\s_k}{A} = u_k\,v_k^T \\ \grad{u_k}{a} &= \fracLR{v_k\k\Im}{\s_k}^T,\quad &\grad{v_k}{a} = \fracLR{\In\k u_k}{\s_k}^T \\ }$$ Let's drop the subscripts and implicitly take $k=\o$ in the following calculations. $$\eqalign{ G &= \grad{\s}{A} \,=\, uv^T \qquad \qquad \{{\rm gradient\;matrix}\} \\ dG &= du\:v^T + u\:dv^T \\ dg &= \c{\vc{dG}} \;\doteq\; \LR{v\k\Im} du + \LR{\In\k u} dv \\ dg &= \fracLR{\LR{v\k\Im}\LR{v\k\Im}^T}{\s} da + \fracLR{\LR{\In\k u}\LR{\In\k u}^T}{\s} da \\ \grad{g}{a} &= \frac{\LR{v\k\Im}\LR{v\k\Im}^T \;+\; \LR{\In\k u}\LR{\In\k u}^T}{\s} \\ }$$ The last line is the requested Hessian (i.e. the second derivative) $$\eqalign{ \Hess{\s}a \,=\, \grad{g}{a} \\ }$$ expressed using vectorized matrices. Without vectorization, it's a $4^{th}$ order tensor.