Could someone provide me with a rigorous proof as to why the derivative of the function $f:t \ni \mathbb{R} \mapsto e^{tA}\in \textrm{Mat}_n (\mathbb{R})$ is $t \mapsto A\cdot e^{tA}$ ? I didn't understand the "elementwise" arguments, as to why the above should hold and when trying to evaluate $f'$ by hand I got stuck at evaluating $$ \lim _{h\rightarrow 0} \frac{||e^{xA}\cdot e^{hA} - e^{xA} -A\cdot e^{xA} \cdot h||}{|h|},$$
where $||\cdot ||$ denotes any norm on $\textrm{Mat}_n (\mathbb{R}) $ that is multiplicative (so that $( \textrm{Mat}_n (\mathbb{R}) , || \cdot ||)$ becomes a Banach algebra) - which is what I have do to, I think (please correct me, if I'm wrong, or using an unnecessary abstract level of discourse), because in the setting of matrix-valued functions the derivative of a matrix becomes the derivative between the Banach spaces $\mathbb{R}$ and $\textrm{Mat}_n (\mathbb{R}) $ (using an isomorphism $\phi:\textrm{Mat}_n (\mathbb{R}) \rightarrow \mathbb{R}^{n^2} $ to do the derivative there and then transporting everything back to $\textrm{Mat}_n (\mathbb{R}) $ seems rather ugly to me - although I tried to do it this way and failed).
Could I replace $\mathbb{R}$ with $\mathbb{C}$ ?
You can begin by recalling that there is another definition for the derivative, namely thinking of maps $\mathbb{R}^m\to\mathbb{R}^n$, one has that the derivative, if it exists, is given by $\displaystyle \lim_{h\to0}\frac{f(x+h)-f(x)}{h}$. Consequently, the derivative of $\text{exp}(xA)$ should be (we know it exists since $\exp(xA)$ is given by a nice power series) $\displaystyle \lim_{h\to0}\frac{\exp((x+h)A)-\exp(xA)}{h}$. Now, the important thing to notice is that $\exp((x+h)A)=\exp(xA+hA)$ which (recalling that $[X,Y]=0$ implies $\exp(X+Y)=\exp(X)\exp(Y)$) is equal to $\exp(xA)\exp(hA)$ and so the derivative of $\exp(xA)$ should be equal to
$\displaystyle \begin{aligned}\lim_{h\to0}\frac{\exp(xA)\exp(hA)-\exp(xA)}{h} &=\exp(xA)\lim_{h\to0}\frac{\exp(hA)-I}{h}\\ &= \exp(xA)\lim_{h\to0 }\frac{1}{h}\left((I+hA+O(h^2))-I\right)\\ &= \exp(xA)\lim_{h\to0}\left(A+O(h)\right)\\ &= \exp(xA)A\end{aligned}$