Considering that $M_{n}(\mathbb{R})$ can be identified with $\mathbb{R}^{n^{2}}$, define $f(X) = X^2$. I want to prove that $f$ is a smooth function.
I don't know how can I relate the matrices with the concept of differentiability in Euclidean space. For example, consider $X \in \mathbb{R}^{n^{2}}$, we can think of $X$ like a matrix, thus $f(X) = X^{2}$ is a matrix too, but similar to what we did earlier, we can consider $X^{2}$ as a vector in $\mathbb{R}^{n^{2}}$, then is it correct to consider $f$ as $$f(X) = (f_{1}(X), \cdots, f_{n^{2}}(X)),$$ where $f_{i}(X)$ is some entry of matrix $X^{2}$ (and therefore is a polynomial)?
Once you identify $M_n(\Bbb R)$ with $\Bbb R^{n^2}$ (in the "natural way"), you can define the projection map $$\pi_{i,j}:M_n(\Bbb R)\to\Bbb R$$ which sends a matrix to its $(i, j)$-th entry. You can also show that this map is a $C^\infty$ function.
Now, as you noted, each component $f_i(X)$ is a polynomial in the entries of $X$. That is, you can write it as a product/sum of some projection maps. Since the product/sum of $C^\infty$ functions is again $C^\infty$, you see that $f_i$ is a $C^\infty$ function.
In turn, that gives you that $f = (f_1, \ldots, f_{n^2})$ is.