The system I am trying to solve is of this form
$x' =ax+5y$
$y' =ay+2z$
$z' =az$
so finding eigen values we solve
$ \begin{bmatrix} a-\lambda &5 &0 \\ 0 &a-\lambda &2 \\ 0 &0 &a-\lambda \end{bmatrix} $
and we will get $\lambda_{1,2,3}=a$
How do I continue on from here? Will the general solution be of the form
$Y = c_1K_1e^{at}+tc_2K_2e^{at}+t^2c_1K_3e^{at}$, where $K_1=K_2=K_3$ are the same eigen vectors since same eigen values
Note then $K_1=K_2=K_3 = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$ , which isn't what the answer in the book is :(
Here's a guide on how to deal with repeated eigenvalues. The idea is to use generalized eigenvectors such that
$$ \begin{aligned} (\textbf{A}-\lambda\textbf{I})\vec{v}_1 &= \vec{0} \\ (\textbf{A}-\lambda\textbf{I})\vec{v}_2 &= \vec{v}_1 \\ (\textbf{A}-\lambda\textbf{I})\vec{v}_3 &= \vec{v}_2 \end{aligned} \tag{1} $$
EDIT: I made a mistake in my initial answer. The third linearly independent solution isn't a straightforward combination of the powers of $t^ne^{\lambda t}$. However, we can still apply the same technique by generalizing the constants
$$ \vec{r}_3 = \big(\vec{v}_1t^2 + \vec{v}_2 Bt + \vec{v}_3 C\big) e^{\lambda t} $$
Plugging this in, we obtain
\begin{align} \vec{r}_3' &= \big(2\vec{v}_1t + \lambda\vec{v}_1 t^2 + B\vec{v}_2 + \lambda B \vec{v}_2t + \lambda C \vec{v}_3\big)e^{\lambda t} \\ &= \big(\lambda \vec{v_1}\big) t^2 e^{\lambda t} + \big(2 \vec{v}_1 + \lambda B \vec{v}_2 \big)t e^{\lambda t} + \big(B\vec{v}_2 + \lambda C\vec{v}_3 \big) e^{\lambda t} \end{align}
$$ \implies \left\{ \begin{aligned} \textbf{A}\vec{v}_1 &= \lambda \vec{v}_1 \\ B\textbf{A}\vec{v}_2 &= 2 \vec{v}_1 + \lambda B \vec{v}_2 \\ C\textbf{A}\vec{v}_3 &= B\vec{v}_2 + \lambda C\vec{v}_3 \end{aligned} \right. \implies \left\{ \begin{aligned} \big(\textbf{A} - \lambda \textbf{I}\big)\vec{v}_1 &= \vec{0} \\ \big(\textbf{A} - \lambda \textbf{I}\big)\vec{v}_2 &= \frac{2}{B} \vec{v}_1 \\ \big(\textbf{A} - \lambda \textbf{I}\big)\vec{v}_3 &= \frac{B}{C}\vec{v}_2 \end{aligned} \right. $$
To get to the same form as $(1)$, we set $B=C=2$, arriving at the general solution in $(2)$.
$$ \begin{bmatrix}x \\ y \\ z \end{bmatrix} = c_1\vec{v}_1e^{\lambda t} + c_2\big(\vec{v}_1te^{\lambda t} + \vec{v}_2e^{\lambda t}\big) + c_3 \big(\vec{v}_1t^2e^{\lambda t} + \vec{v}_2 2te^{\lambda t} + \vec{v}_3 2 e^{\lambda t}\big) \tag{2} $$
Let's go on to find these vectors. Note the product first $$ (\textbf{A}-\lambda\textbf{I})\vec{v} = \begin{bmatrix} 0 & 5 & 0 \\ 0 & 0 & 2 \\ 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 5y \\ 2z \\ 0 \end{bmatrix} $$
The first eigenvector satisfies
$$ (\textbf{A}-\lambda\textbf{I})\vec{v}_1 = \begin{bmatrix} 5y_1 \\ 2z_1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} $$
which gives $y_1 = z_1 = 0$. There is no restriction on $x_1$, so we can pick any value we want. Note that we need $x_1 \ne 0$ to avoid a trivial solution, so let's say $x_1 = 1$
Then, the first generalized eigenvector must satisfy
$$ (\textbf{A}-\lambda\textbf{I})\vec{v}_2 = \begin{bmatrix} 5y_2 \\ 2z_2 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$
This time, $y_2=1/5$, $z_2=0$, and again any value for $x_2$ will work. We pick $x_2=0$ since there's no possibility of getting a zero vector.
You can do the last case. The 3 vectors are
$$ \vec{v}_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \quad \vec{v}_2 = \begin{bmatrix} 0 \\ 1/5 \\ 0 \end{bmatrix}, \quad \vec{v}_3 = \begin{bmatrix} 0 \\ 0 \\ 1/10 \end{bmatrix} $$
Plugging in $(2)$, we get the full solution $$ \begin{bmatrix}x \\ y \\ z \end{bmatrix} = \begin{bmatrix} c_1 e^{at} + c_2 te^{at} + c_3t^2e^{at} \\ \frac{1}{5}\big(c_2e^{at} + 2c_3te^{at}\big) \\ \frac{1}{5}c_3e^{at} \end{bmatrix} $$