I would like to differentiate the induced norm $\|I - \gamma P\|_2$ with respect to $\gamma$, where $I$ and $P$ are $n\times n$ matrices and $\gamma$ is a scalar. How do I proceed?
The Matrix Cookbook doesn't seem to address this. This question seems related, but I think it's unlikely that the result generalizes to matrices.
Thanks in advance.
$\def\p#1#2{\frac{\partial #1}{\partial #2}}\def\pp#1#2{\frac{d#1}{d#2}}$ The induced ${\tt2}$-norm is identical to the Schatten ${\infty}$-norm (also known as the spectral norm).
The spectral norm of $A$ can be written in terms of its SVD $$\eqalign{ A &= USV^T = \sum_{k=1}^{rank(A)} \sigma_k u_k v_k^T\\ \|A\| &= \sigma_1 \qquad\big(\sigma_1\ge\sigma_2\ge\ldots\ge\sigma_n\ge 0\big) \\ }$$ and its differential and gradient can be written using the first columns of $U$ and $V$ $$\eqalign{ d\,\|A\| &= u_1^T\,dA\;v_1 \\ \p{\|A\|}{A} &= u_1v_1^T \\ }$$ Setting $\,A=\gamma P-I\;$ yields the desired derivative $$\eqalign{ d\,\|A\| &= u_1^T\big(P\,d\gamma\big)\,v_1 \\ \pp{\|A\|}{\gamma} &= u_1^TPv_1 \\ }$$