Below is my attempt to solve the problem. However, my answer is different from the book's answer. I would like to know where I went wrong or maybe the book's answer is wrong.
Bob
Problem:
Solve the following system of differential equations.
\begin{eqnarray*}
5x' + y' - 3x + y &=& 0 \\
4x' + y' - 3x = -3t \\
\end{eqnarray*}
Answer:
To solve this system of differential equations we will use the operator method.
\begin{eqnarray*}
(5D-3)x + (D+1)y &=& 0 \\
(4D-3)x + Dy &=& -3t \\
D(5D-3)x + D(D+1)y &=& 0 \\
(D+1)(4D-3)x + D(D+1)y &=& (D+1)(-3t) = -3t - 3 \\
D(5D-3)x - (D+1)(4D-3)x &=& 3t + 3 \\
(5D^2 - 3D)x - (4D^2 + D - 3)x &=& 3t + 3 \\
(D^2 - 4D + 3)x &=& 3t + 3 \\
\end{eqnarray*}
Now to solve this equation we use the method of undermined coefficients.
\begin{eqnarray*}
x &=& x_c + x_p \\
m^2 - 4m + 3 &=& 0 \\
(m-3)(m-1) &=& 0 \\
m = 3 &\text{ or }& m = 1 \\
x_c &=& c_1e^{t} + c_2e^{3t} \\
x_p &=& At + B \\
x'_p &=& A \\
-4A + 3(At+B) &=& 3t + 3 \\
3At + 3B - 4A &=& 3t + 3 \\
3A &=& 3 \\
A &=& 1 \\
3B - 4A &=& 3 \\
3B - 4(1) &=& 3 \\
3B 4 &=& 3 \\
B &=& \frac{7}{3} \\
x &=& c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} \\
\end{eqnarray*}
Now we need to find $y$.
\begin{eqnarray*}
(4D-3)x + Dy &=& -3t \\
(4D-3)(c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} ) + Dy &=& -3t \\
4c_1e^t + 12c_2e^{3t} + 4 - 3c_1e^{t} - 3c_2e^{3t} - 3t - 7 + Dy &=& -3t \\
c_1e^t + 12c_2e^{3t} + 4 - 3c_2e^{3t} - 3t - 7 + Dy &=& -3t \\
c_1e^t + 9C_2e^{3t} - 3 + Dy &=& 0 \\
\end{eqnarray*}
\begin{eqnarray*}
\frac{dy}{dt} &=& 3 - c_1e^t - 9c_2e^{3t} \\
y &=& 3t - c_1e^t - 3c_2e^{3t} + c_3\\
\end{eqnarray*}
Now we need to eliminate the constant $c_3$.
\begin{eqnarray*}
x' &=& c_1e^{t} + 3c_2e^{3t} + 1 \\
5(c_1e^{t} + 3c_2e^{3t} + 1) + y' - 3x + y &=& 0 \\
5(c_1e^{t} + 3c_2e^{3t} + 1) + 3 - c_1e^t - 9c_2e^{3t} - 3x + y &=& 0 \\
5c_1e^{t} + 15c_2e^{3t} + 5 + 3 - c_1e^t - 9c_2e^{3t} - 3( c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} ) + y &=& 0 \\
5c_1e^{t} + 15c_2e^{3t} + 8 - c_1e^t - 9c_2e^{3t} - 3( c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} ) + y &=& 0 \\
4c_1e^{t} + 6c_2e^{3t} + 8 - 3( c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} ) + y &=& 0 \\
c_1e^{t} + 3c_2e^{3t} + 8 - 3t - 7 + 3t - c_1e^t - 3c_2e^{3t} + c_3 &=& 0 \\
c_1e^{t} + 3c_2e^{3t} + 8 - 7 - c_1e^t - 3c_2e^{3t} + c_3 &=& 0 \\
3c_2e^{3t} + 1 - 3c_2e^{3t} + c_3 &=& 0 \\
c_3 &=& -1 \\
\end{eqnarray*}
Hence the answer is:
\begin{eqnarray*}
y &=& 3t - c_1e^t - 3c_2e^{3t} - 1 \\
\end{eqnarray*}
The books answer is:
\begin{eqnarray*}
x &=& c_1e^{t} + c_2e^{3t} + t + \frac{7}{3} \\
y &=& -c_1e^{t} - 3c_2e^{3t} + 3t + 1 \\
\end{eqnarray*}
Your answer is the right one.
Checked with Mathematica by direct substitution:
Your solution looks right to me as well, but I didn't go into the details too much. At least I see no mistake in the part where you find $c_3$.
In the same way, we can see that the book's answer is wrong: