Let $A\in\mathbb{R}^{n\times n}$. I want to show that $e^A=\sum_{k=0}^{\infty}\frac{A^k}{k!}$ converges, so let's define $S\left(n\right)=\sum_{k=0}^n\frac{A^k}{k!}$.
To show: $\lim_{n\rightarrow\infty}S\left(n\right)=S$
In general metric spaces, the convergence would mean the following: $$ \forall\varepsilon>0:\ \exists M\in\mathbb{N}:\ n\ge M\ \Longrightarrow\ d\left(S\left(n\right){,}\ S\right)<\varepsilon$$
But in case of matrices, what does $d\left(S\left(n\right){,}\ S\right)$ mean? What is the distance that I need to consider to show the convergence?
The space $\Bbb R^{n\times n}$ is a finite-dimensional real vector space, and therefore there are many norms that you can define in it. For each such norm $\|\cdot\|$, you can consider the distance $d(A,B)=\|A-B|$. Since all the norms are equivalent, any of them will do. For instance, you can consider the norm$$\bigl\|(a_{ij})_{1\leqslant i,j\leqslant n}\bigr\|=\sqrt{\sum_{i,j=1}^n|a_{ij}|^2}.$$