Let the following homogenous system of differential equation:
$$\left\{\begin{array}{l}x'=ax+by\\y'=cx+dy\end{array}\right.$$
Let $A = \begin{pmatrix}a&b\\c&d\end{pmatrix}$ and $A$ is not diagonalizable and has complex roots conjugate: $\alpha\pm i\beta$
How do I arrive at the fundamental Matrix of solutions:
$$e^{\alpha t}\begin{pmatrix}\cos\beta t&\sin\beta t\\\sin\beta t&\cos\beta t\end{pmatrix}$$
Why this particular matrix?
Pretend for the moment that you’re working with complex matrices. The coefficient matrix has two distinct eigenvalues and you can find corresponding complex eigenvectors $v_1$ and $v_2$ using the usual method, but working in the complex domain. Plunging ahead, you then have $$e^{tA} = \begin{bmatrix}v_1\\v_2\end{bmatrix} \begin{bmatrix}e^{(\alpha+i\beta)t}&0\\0&e^{(\alpha-i\beta)t}\end{bmatrix} \begin{bmatrix}v_1\\v_2\end{bmatrix}^{-1}.$$ Multiply this out and simplify using the identities $$\cos\theta = \frac12\left(e^{i\theta}+e^{-i\theta}\right) \\ \sin\theta = \frac1{2i}\left(e^{i\theta}-e^{-i\theta}\right).$$