Let's consider the DE $x'=Ax$ where $$A=\begin{pmatrix}0&0&0\\1&0&-1\\0&0&-1\end{pmatrix}$$
Now, in one of my textbooks it says that for each eigenvalue $\lambda$ with algebraic multiplicity $k$ we get $k$ elements of the fundamental system via $$\begin{array} a e^{\lambda t}v_1, & e^{\lambda t}(v_2 +tv_1),&\dots &\end{array} e^{\lambda t}(v_k+\dots+t^{k-1}v_1)$$where the $v_1,\dots,v_n$ are the generalized eigenvectors corresponding to $\lambda$.
If I use this statement I first get the generalized eigenvectors via $\chi_A(\lambda)=-\lambda^2(1+\lambda)$ to be
- $u_1=(0,1,1)$ for $\lambda_1=-1$.
- For $\lambda_2=0$ I chose $v_1=(1,0,0)\in\ker(A^2)$ and got $v_2=Av_1=(0,1,0)^T$.
This should yield the fundamental system $$\left\{(0,1,1)e^{-t},(1,0,0),(t,1,0)\right\}$$which doesn't produce solutions if I'm not mistaken. Where is the error?
A Jordan block with eigenvalue $\lambda$ and multiplicity $n$ for $A$ corresponds to a cyclic subspace generated by a vector $x\ne 0$ with basis $$ \{ x, (A-\lambda I)x, (A-\lambda I)^2x,\cdots,(A-\lambda I)^{n-1}x \}. $$ And $(A-\lambda I)^nx=0$, but $(A-\lambda I)^{n-1}\ne 0$. On this cyclic subspace, $e^{tA}$ is represented by $$ e^{tA}x= e^{\lambda t}e^{t(A-\lambda I)}x \\ = e^{\lambda t}\left(x+t(A-\lambda I)x+\frac{t^2}{2!}(A-\lambda I)^2x+\cdots+\frac{t^{n-1}}{(n-1)!}(A-\lambda I)^{n-1}x\right) $$ You have cyclic subspaces for $\lambda=0,1$. The one for $\lambda=-1$ is one-dimensional. The one for $\lambda=0$ is two-dimensional. $\lambda=-1$ gives solution $$ e^{-t}\begin{pmatrix}0 \\ 1 \\ 1\end{pmatrix}. $$ $\lambda=0$ gives one solution that is constant in t, and another that is linear in $t$: $$ \begin{pmatrix}0 \\ 1 \\ 0\end{pmatrix}, \begin{pmatrix}1\\0\\0\end{pmatrix}+t\begin{pmatrix}0\\1\\0\end{pmatrix}. $$ You can directly check that all three of these satisfy the equation $\frac{d}{dt}x(t)=Ax(t)$, and the initial values of these solutions are vectors that span $\mathbb{C}^3$. So linear combinations of these three solutions form a basis of all solutions of $\frac{dx}{dt}=Ax(t)$. You formed incorrect combinations based on the statement that you quoted. It's not valid to multiply the first of these last two by $t$ and then subtract this from the second, because multiplication by $t$ changes the first into a non-solution.