Let $V$ be a finite dimensional vector space over $\mathbb{R}$. Let $\mbox{GL}(V)\subset\mbox{End}(V)$ denote the subset of invertible maps.
Let $$m:\mbox{GL}(V)\times \mbox{GL}(V)\longrightarrow \mbox{GL}(V)$$ denote the "composition" map defined by $$(A,B)\mapsto m(A,B)=A\circ B.$$
- Find the linear approximation to $m$ at $(A,B)\in \mbox{GL}(V)\times \mbox{GL}(V)$.
- Give a formula for the derivative of $m$ at $(A,B)\in \mbox{GL}(V)\times \mbox{GL}(V)$.
My attempt and questions: We know that $\mbox{GL}(V)$ is an open subset of $\mbox{End}(V)$, therefore $\mbox{GL}(V)\times \mbox{GL}(V)$ is an open subset of $\mbox{End}(V)\times \mbox{End}(V)$, also we know that $\mbox{End}(V)\times \mbox{End}(V)$ is a vector space.
Therefore, for $(\tilde{A},\tilde{B})\in \mbox{GL}(V)\times \mbox{GL}(V)$ we have $$m(\tilde{A},\tilde{B})=A\circ B + A\circ (\tilde{A}-A)+(\tilde{B}-B)\circ B+(\tilde{A}-A)\circ (\tilde{B}-B).$$ Then, for $(\tilde{A},\tilde{B})$ sufficiently close to $(A,B)$ we have $$m(\tilde{A},\tilde{B})\thickapprox A\circ B + A\circ (\tilde{A}-A)+(\tilde{B}-B)\circ B.$$ Is this the linear approximation to $m$ at $(A,B)\in \mbox{GL}(V)\times \mbox{GL}(V)$?
In this item, we know that $T_{(A,B)}\left(\mbox{GL}(V)\times \mbox{GL}(V)\right)=\mbox{End}(V)\times \mbox{End}(V)$ and $T_{m(A,B)}\mbox{GL}(V)=\mbox{End}(V)$. Therefore, given $(K,S)\in T_{(A,B)}\left(\mbox{GL}(V)\times \mbox{GL}(V)\right)$, i. e, $K, S \in \mbox{GL}(V)$, let $c(t):=(A+tK,B+tS)$ be a curve, note that $c(0)=(A,B)$ and $c'(0)=(K,S)$. Therefore, $$\begin{array}{rcl}\left.Dm\right|_{(A,B)}(K,S)&=&\left.\frac{d}{dt}\right|_{t=0}m(c(t))=\left.\frac{d}{dt}\right|_{t=0}m(A+tK,B+tS) \\ &=& \left.\frac{d}{dt}\right|_{t=0}\left(A\circ B+t K\circ B+t A\circ S+t^{2} K\circ S\right) \\ &=&K\circ B + A \circ S.\end{array}$$ Therefore, the formula for the derivative of $m$ at $(A,B)$ is $$\begin{array}{rcl} \left.Dm\right|_{(A,B)}:T_{(A,B)}\left(\mbox{GL}(V)\times \mbox{GL}(V)\right) &\rightarrow & T_{m(A,B)}\mbox{GL}(V) \\ (K,S) &\mapsto & K\circ B + A \circ S. \end{array}$$ Is my answer correct?
Your calculation is correct. To be extremely picky, you need to justify why $A+ tK$ is a curve in $GL(V)$. In fact it might not be for large $t$, but the continuity of $\det$ implies that $A+tK$ is invertible for $t$ small and this is what we really need.
Note that more or less the same can be said for a general Lie group $G$: The multiplication mapping $m :G\times G\to G$ has derivative
$$ (dm)_{(g_1, g_2)} (X_1, X_2) =(r_{g_2})_*X_1 + (\ell_{g_1})_* X_2,$$
where $\ell_g, r_g$ denote left and right multiplication respectively. The proof is quite trivial, once we recall that $dm$ is bilinear, as suggested by Ivo in the comment.