Consider the linear system
$$ \frac{dY}{dt} = \begin{pmatrix} 1 & -1 \\ 1 & 3 \\ \end{pmatrix} Y $$
(a) Show that the function
$$ Y(t) = \begin{pmatrix} te^{2t} \\ -(t + 1)e^{2t}\\ \end{pmatrix} $$
is a solution to the differential equation.
I verified this without trouble. The second part I am stuck on:
(b) Solve the initial-value problem
$$ \frac{dY}{dt} = \begin{pmatrix} 1 & -1 \\ 1 & 3 \\ \end{pmatrix} Y \text{, where } Y(0) = \begin{pmatrix} 0 \\ 2 \\ \end{pmatrix} $$
The solution of a system $$ \dfrac{d}{dt}y=Ay $$ with initial conditions $y(0)=y_0$ (where $y$ is a vector function and $A$ is a matrix) is:
$$ y= e^{At}y_0 $$ in your case we have:
$$ e^{At}= e^{2t} \begin{bmatrix} 1-t&-t\\ t&1+t \end{bmatrix} $$ so the solution is: $$ e^{At}y(0)= e^{2t} \begin{bmatrix} 1-t&-t\\ t&1+t \end{bmatrix} \begin{bmatrix} 0\\ 2 \end{bmatrix} = \begin{bmatrix} -2te^{2t}\\ 2(1+t)e^{2t} \end{bmatrix} $$
To calculate the exponetial of a matrix see the answers in: Exponential of matrix