I am interested in calculating quantities such as
$$\frac{\operatorname{d}}{\operatorname{d}\!t} e^{A(t)},\quad \frac{\operatorname{d}}{\operatorname{d}\!t} e^{\int_{a}^{t}A(s)\operatorname{d}\! s}$$
but I am having a lot of trouble finding sources that describe how matrix differentiation and integration are performed. Every source I read seems to only describes the case when $A(t)$ is of the form $tA$ where $A$ is a constant matrix instead of the more general case.
So far, I have concluded that I can safely differentiate under the summation sign, then assuming the fundamental theorem of calculus holds,
$$\frac{d}{dt} e^{\int_{a}^{t}A(s)ds} = \sum_{n=0}^{\infty} \frac{d}{dt}\frac{ \left(\int_{a}^{t}A(s)ds\right)^{n}}{n!}$$
but then, I am led to the question whether or not $A(t) \int_{a}^{t}A(s)ds = \int_{a}^{t}A(s)ds A(t)$ to proceed further.
EDIT: Based on the comments of Stephen Montgomery-Smith, I can see that if $AA' = A'A$, then
$$\frac{d}{dt}e^{A(t)} = \sum_{n=0}^{\infty} \frac{d}{dt}\frac{(A(t))^{n}}{n!} = \sum_{n=0}^{\infty}\frac{(A(t))^{n-1}}{(n-1)!}A'(t) = \sum_{n=0}^{\infty}\frac{(A(t))^{n-2}}{(n-1)!}A'(t)A = \dots $$
$$= A^{\prime}(t) \sum_{n=0}^{\infty}\frac{(A(t))^{n-1}}{(n-1)!}= A^{\prime}(t)e^{A(t)}$$
and using this, I can be led to believe that the problem with these derivates are solely related to their commutativity (Is this correct?). My only problem that remains in that case is a question of the chain rule: When I wrote above,
$$\frac{d}{dt}\frac{(A(t))^{n}}{n!} = \frac{(A(t))^{n-1}}{(n-1)!}A'(t),$$
Is this the correct form of the chain rule for matrix valued functions? That is, is $$\frac{d}{dt} A(B(t)) = A^{\prime}(B(t)) B^{\prime}(t)?$$
I can only guess that this is true because otherwise if
$$\frac{d}{dt} A(B(t)) = B^{\prime}(t) A^{\prime}(B(t))$$
then $\frac{d}{dt} e^{A(t)} = A^{\prime}(t)e^{A(t)}$ without any requirement like $A^{\prime}A = AA^{\prime}$.
The derivative $A'(t) = \frac{\operatorname{d}}{\operatorname{d}\!t}A(t)$ of a matrix function $A(t)$ equals the matrix of derivatives of the components
$$(\frac{\operatorname{d}}{\operatorname{d}\!t}A(t))_{ij} = \frac{\operatorname{d}}{\operatorname{d}\!t}(A(t)_{ij}).$$
You may take this as definition. Calculating in components gives
$$\frac{\operatorname{d}}{\operatorname{d}\!t}A(t)^2 = A'(t) A(t) + A(t) A'(t),$$
which generalizes as
$$\frac{\operatorname{d}}{\operatorname{d}\!t}A(t)^n = \sum_{i=1}^{n} A^{i-1}(t) A'(t) A^{n-i}(t).$$
In case $[A(t), A'(t)]= 0$ the latter formula reduces to
$$\frac{\operatorname{d}}{\operatorname{d}\!t}A^n(t) = nA'(t)A^{n-1}(t) = n A^{n-1}(t)A'(t),$$
which allows to compute
$$\frac{\operatorname{d}}{\operatorname{d}\!t} (exp \ A(t)) = A'(t) \ exp \ A(t) = (exp \ A(t)) \ A'(t)$$
by differentiating under the summation sign.
Note. 1. I do not know whether a closed expression $\frac{\operatorname{d}}{\operatorname{d}\!t} (exp \ A(t))$ exists for arbitrary A(t).