Reference Request: Differentials of Operators

Question

Consider, for example, the map $f: \mathbb{R}^{n \times n} \rightarrow \mathbb{R}^{n \times n}, f(A) = A^2.$ Then its differential is $df(A)(T) = AT+TA$. I would like a reference that states what this differential means and then how to obtain such results, but not necessarily in a completely rigorous way. I also understand that differentials can be defined and manipulated in the usual way for functionals (e.g. for the Lagrangian, leading to the Euler-Lagrange equations) and I'd like to see this done without developing the whole machinery of variational calculus.

In short, I'm looking for a clear treatment of differentials of operator-valued functions. I've tried looking up books on matrix calculus, calculus on normed vector spaces and variational calculus but haven't found anything suitable (the closest option was Cartan's Differential Calculus, but I'd like something more concrete). Where do people learn this sort of thing?

Answer 1

Just compute the directional derivative, as you would in ordinary calculus. $df(A)(T) = \lim\limits_{h\to 0} \dfrac{f(A+hT)-f(A)}h$. Just do the matrix computation: \begin{align*} \frac{f(A+hT)-f(A)}h &= \frac{(A+hT)^2-A^2}h = \frac{h(AT+TA) + h^2T^2}h \\ &= (AT+TA) + hT^2 \to AT+TA \quad\text{as}\quad h\to 0. \end{align*} The point is that it's nothing different from calculus in Euclidean space, since the space of matrices is naturally a finite-dimensional Euclidean space.

Aside from other texts mentioned, Dieudonné's Treatise on Analysis is a standard reference. Differential Calculus in normed spaces appears in Volume 1.

Answer 2

The right setting to talk about differentiability is the notion of a normed vector space. For example real $n\times n$ matrices are (obviously) a vector space, moreover you can introduce a norm on it. Also functionals in calculus of variations can often be written as a function between two normed vector spaces (the source being some vector space of functions, the target being the real numbers).

However, I'd recommend to start with something a bit simpler – learning how this formalism works in Euclidean spaces – and then learning the topic in more specialized contexts.

I'd recommend any of the following books:

• W. Rudin's Principles of mathematical analysis,
• T. Shifrin's Multivariable mathematics,
• M. Spivak's Calculus on manifolds.

(Edit...) and these online materials:

• Introduction to Manifolds from Oxford,
• Multivariable calculus from Bristol,
• Ted Shifrin's lectures on YouTube. In the context of the posed problem Lectures 21 and 22 are especially relevant.

Answer 3

The total derivative of a differential map $f\colon \Omega \subseteq \Bbb R^n \to \Bbb R^k$ at a point $x \in \Omega$, where $\Omega$ is open, is the unique linear map $Df(x)$ such that $$\lim_{h \to 0} \frac{f(x+h)-f(x)- Df(x)(h)}{\|h\|} = 0. $$Since matrix spaces are identified with Euclidean spaces themselves, it makes sense to compute derivatives of maps between matrix spaces. For instance, we have the chain rule $D(g\circ f)(x) = Dg(f(x))\circ Df(x)$, the total derivative of a linear map is itself, and if $B\colon \Bbb R^n \times \Bbb R^m \to \Bbb R^p$ is bilinear, its derivative is given by $$DB(x,y)(h,k) = B(x,k) + B(h,y).$$In your case, we can write $f(A) = A^2$ as $f(A) = g(\Delta(A))$, where $\Delta(A)= (A,A)$ is the (linear) diagonal map and $g(A,B) = AB$ is bilinear. So $$\begin{align} Df(A)(T) &= D(g\circ \Delta)(A)(T) = Dg(A,A) \circ D\Delta(A)(T) \\ &= Dg(A,A)(T,T) = g(A,T)+g(T,A) \\ &= AT+TA, \end{align}$$as wanted.

Answer 4

A book I've had for a long time (I think I purchased it from a university bookstore in 1981 or 1982) might be helpful. Although it's a bit weak on specific examples, the exposition is very straightforward and is accessible to someone with a fairly limited background (much less than for standard functional analysis texts, except maybe for Kreyszig's Introductory Functional Analysis with Applications, which might also be worth looking at). I'm including the contents because not much specific seems to be posted on the internet about it. Indeed, the only mention in Stack Exchange that I could find is this 4 November 2013 comment by me.

Leopoldo Nachbin, Introduction to Functional Analysis: Banach Spaces and Differential Calculus, translation of the 1976 Portuguese edition by Richard Martin Aron, Monographs and Textbooks in Pure and Applied Mathematics #60, Marcel Dekker, 1981, xii + 166 pages. Amer. Math. Monthly review

CONTENTS (pp. v-vi). PREFACE (pp. vii-ix).

PART I. BANACH SPACES (pp. 1-84).

1. Normed Spaces (pp. 3-9). 2. Banach Spaces (pp. 10-19). 3. Normed Subspaces (pp. 20-24). 4. Equivalent Norms (pp. 25-32). 5. Spaces of Continuous Linear Operators (pp. 33-42). 6. Continuous Linear Forms (pp. 43-49). 7. Isometries (pp. 50-51). 8. Cartesian Products and Direct Sums (pp. 52-56). 9. Cartesian Products of Normed Spaces (pp. 57-59). 10. Topological Direct Sums (pp. 60-62). 11. Finite Dimensional Normed Spaces (pp. 63-76). 12. Spaces of Continuous Multilinear Operators (pp. 77-84).

PART II. DIFFERENTIAL CALCULUS (pp. 85-160).

13. Differential Calculus in Normed Spaces (pp. 87-91). 14. The Differential in Normed Spaces (pp. 92-96). 15. Continuous Affine Tangent Mappings (pp. 97-98). 16. Some Rules of Differential Calculus (pp. 99-111). 17. The Scalar Variable Case (pp. 112-114). 18. The Lagrange Mean Value Theorem (pp. 115-123). 19. Mappings with Zero or Constant Derivatives (pp. 124-126). 20. Interchanging the Order of Differentiation and Limit (pp. 127-130). 21. Continuously Differentiable Mappings (pp. 131-132). 22. Partial Differentiation (pp. 133-142). 23. Natural Identifications for Multilinear Mappings (pp. 143-149). 24. Higher Order Differentiation (pp. 150-160).

NOTATION (pp. 161-162). BIBLIOGRAPHY (pp. 163-164). INDEX (pp. 165-166).