Let $M(n \times n, \mathbb{R})$ denote the set of $n \times n$ matrices over $\mathbb{R}$ and $S(n \times n, \mathbb{R}) \subset M(n \times n, \mathbb{R})$ the subspace of matrices that are symmetric.
Define $f: M(n \times n, \mathbb{R}) \rightarrow S(n \times n, \mathbb{R})$ by $A \mapsto A A^T$. In Hamilton's Mathematical Gauge Theory he says the differential of $f$ at a point $A \in O(n) = f^{-1}(I)$ in the direction $X \in M(n \times n, \mathbb{R})$ is $$(D_Af)(X) = XA^T + AX^T.$$
I am familiar with the notion of differentials of maps between manifolds, but I cannot see how he obtained the above expression. I have two questions.
- How did he compute the differential?
- What exactly does he mean by "in the direction of $X$"? If he means identifying the differential with the directional derivative, shouldn't $X$ be a tangent vector? Or is he implicitly identifying $M(n \times n, \mathbb{R})$ with its tangent space at $A$?
Regarding your 2nd question: the tangent space of the space of matrices $M(n\times n, \mathbb R)$ at a point $A$ is, as you said, identified with $M(n\times n, \mathbb R)$ itself.
Regarding your 1st question: you may view the tangent vectors as velocities of curves $c:\mathbb R\to M(n\times n, \mathbb R)$ starting at $A$, i.e. the tangent vector $X=c’(0)$, where $c(t)\in M(n\times n,\mathbb R)$ and $c(0)=A$. Then we have
$$ D_Af(X)= \frac{d}{dt} |_{t=0} f(c(t))=\frac{d}{dt} |_{t=0} c(t)c(t)^T = c’(0)c(0)^T+c(0)c’(0)^T= XA^T+AX^T.$$