$\mathbf {Consider \space the \space system}$ $$\frac{dx}{dt} = -x-6y $$
$$\frac{dy}{dt} = 3x+5y$$
$\mathbf {Find\space the\space general\space solution\space of\space the \space system.}$
For this problem, I solved eigenvalues, which are $ 2 \pm 6 \mathbf i $ and got the general solution: $$x(t) = k_1e^{2t}[2cos\left(6t\right)]+ k_2e^{2t}[2sin\left(6t\right)]$$ $$y(t) = k_1e^{2t}[-cos(6t)+2sin(6t)]+k_2e^{2t}[-2cos(6t)-sin(6t)]$$
But, I am not sure if this is correct. Please help me to check this answer.
Use Laplace Transform:
$$x'(t)=-x(t)-6y(t)\Longleftrightarrow$$ $$\mathcal{L}_{t}\left[x'(t)\right]_{(s)}=\mathcal{L}_{t}\left[-x(t)-6y(t)\right]_{(s)}\Longleftrightarrow$$ $$sx(s)-x(0)=-x(s)-6y(s)\Longleftrightarrow$$ $$sx(s)+x(s)=x(0)-6y(s)\Longleftrightarrow$$ $$x(s)\left[s+1\right]=x(0)-6y(s)\Longleftrightarrow$$ $$x(s)=\frac{x(0)-6y(s)}{s+1}$$
$$y'(t)=3x(t)+5y(t)\Longleftrightarrow$$ $$\mathcal{L}_{t}\left[y'(t)\right]_{(s)}=\mathcal{L}_{t}\left[3x(t)+5y(t)\right]_{(s)}\Longleftrightarrow$$ $$sy(s)-y(0)=3x(s)+5y(s)\Longleftrightarrow$$ $$sy(s)-5y(s)=3x(s)+y(0)\Longleftrightarrow$$ $$y(s)\left[s-5\right]=3x(s)+y(0)\Longleftrightarrow$$ $$y(s)=\frac{3x(s)+y(0)}{s-5}$$
So, we can find that:
$$x(s)=\frac{x(0)-6\cdot\frac{3x(s)+y(0)}{s-5}}{s+1}\Longleftrightarrow x(s)=\frac{x(0)(s-5)-6y(0)}{13+s(s-4)}$$
$$y(s)=\frac{3\cdot\frac{x(0)-6y(s)}{s+1}+y(0)}{s-5}\Longleftrightarrow y(s)=\frac{y(0)+3x(0)+y(0)s}{13+s(s-4)}$$
And with the Inverse Laplace Transform, find that:
$$\mathcal{L}_{s}^{-1}\left[x(s)\right]_{(t)}=\mathcal{L}_{s}^{-1}\left[\frac{x(0)(s-5)-6y(0)}{13+s(s-4)}\right]_{(t)}\Longleftrightarrow$$ $$x(t)=\mathcal{L}_{s}^{-1}\left[\frac{x(0)(s-5)-6y(0)}{13+s(s-4)}\right]_{(t)}\Longleftrightarrow$$ $$x(t)=e^{2t}\left(x(0)\cos(3t)-(x(0)+2y(0))\sin(3t)\right)$$
$$\mathcal{L}_{s}^{-1}\left[y(s)\right]_{(t)}=\mathcal{L}_{s}^{-1}\left[\frac{y(0)+3x(0)+y(0)s}{13+s(s-4)}\right]_{(t)}\Longleftrightarrow$$ $$y(t)=\mathcal{L}_{s}^{-1}\left[\frac{y(0)+3x(0)+y(0)s}{13+s(s-4)}\right]_{(t)}\Longleftrightarrow$$ $$y(t)=e^{2t}\left(y(0)\cos(3t)+(y(0)+x(0))\sin(3t)\right)$$
So, our solution is: