I know that we can define a function $f$ over a matrix $A$ by doing the following, we diagonalise $A$ and then apply $f$ to all eigenvalues of $A$. After that we can perform a change of basis if we need our expression to be in a specific basis.
But what happens if the matrix cannot be written in a diagonal form, can I extend this definition? Do you any book that explains this? In $\mathbb{C}$ all matrices can be written in a diagonal form so I am wondering if there is a way to go back to $\mathbb{R}$ after applying $f$ but I am not sure if I am going on the right way. Thanks.
If you begin with a real matrix $A$ and a one-variable real analytic function $f(x), \; $ you do get real $f(A).$
The Jordan part, $P^{-1}AP = J, \;$ is writing $J = D + N$ where $N$ is strictly upper triangular, and, because of the Jordan block description, $DN = ND.$ As a result, we can write $f(J) =f(D+N)$ using all available power series conveniences, along with the fact that $N^n = 0.$ The most common application is $e^{(D+N)t} = e^{Dt} e^{Nt} = e^{Nt} e^{Dt},$ where the entries of $e^{Nt}$ are explicit polynomials in $t.$
The part that I do not see many students writing correctly is just this, turn the thing around, $PJP^{-1} = A,$ finally $P f(J) P^{-1} = f(A).$ In order to do this, it is necessary to confirm that $P P^{-1} = I$ and both $P^{-1}AP = J, \;$ and $P f(J) P^{-1} = f(A).$ It is not enough to know the final Jordan form, you need the actual matrix $P$ and its correct inverse. Here is one from yesterday How to find Jordan basis of a matrix
Reverse direction $PJP^{-1} = A:$
$$ \left( \begin{array}{rrrrr} 1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 1 & 0 & 1 & 0 & 0 \end{array} \right) \left( \begin{array}{rrrrr} -1 & 0 & 0 & 0 & 0 \\ 0 & -1 & 1 & 0 & 0 \\ 0 & 0 & -1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 0 & 1 \end{array} \right) \left( \begin{array}{rrrrr} 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ -1 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 1 & -1 & 0 & 0 \end{array} \right) = \left( \begin{array}{rrrrr} -1 & 0 & 0 & 0 & 0 \\ -1 & 1 & -2 & 0 & 1 \\ -1 & 0 & -1 & 0 & 1 \\ 0 & 1 & -1 & 1 & 0 \\ 0 & 0 & 0 & 0 & -1 \end{array} \right) $$ $$ $$
Worthwhile exercise for the reader: find $e^{At}$