I have to solve the following first-order linear system, $x(t)$ represents one population and the $y(t)$ represents another population that lives in the same ecosystem: (Note: $'$ denotes prime)
\begin{align} x' = -5x - 20y & \text{(Equation 1)} \\ y' = 5x + 7y & \text{(Equation 2)} \end{align}
I start off with finding the derivative of (Equation 1) which gives me:
$$x'' = -5x' - 20 y'$$
and then substitute $y'$ using the equations above (Equation 2):
$$x'' = -5x' - 20 (5x + 7y)$$
foil expansion:
$$x'' = -5x' - 100x - 140y$$
so I insert values for $y$ through reordering an existing equation:
$$y = -\frac{1}{20}(x'+5x)$$
thus \begin{align} x'' = -5x'-100x-140\bigg(-\frac{1}{20}x'-\frac{1}{4}x\bigg) \\ x'' = -5x'-100x+7x'+35x \\ x''+ 5x'+100x-7x'-35x=0 \\ x''-2x'+65x=0 \end{align}
Then put into auxillary form: $r^2-2r+65=0$ thus $r = 1 \pm 8i$ ..etc.
Eventually I get $x(t)$ and $y(t)$. I have to predict what will happen to the population densities over a long time. How can I do this?
Note: The book describes using equilibrium points and determining their stability to do this. Upvotes to whoever shows this specific method. I appreciate any additional explanations however.
Edit: Here is my answer for $x(t)$ and $y(t)$ for clarification purposes. ($C_1$ and $C_2$ are constants)
$x(t) = C_1e^{t} \sin 8t + C_2e^{t} \cos 8t$
$y(t) = \frac{1}{10}e^{t} (4C_2 \sin 8t-4C_1 \cos 8t-3C_1 \sin 8t-3C_2 \cos 8t)$
I agree with your auxiliary result.
You could have also written the system as a matrix and used eigenvalues and eigenvectors to solve it.
Using the system matrix approach, we end up finding the solutions (these do not match yours):
$$x(t)=\frac{1}{4} c_1 e^t (4 \cos (8 t)-3 \sin (8 t))-\frac{5}{2} c_2 e^t \sin (8 t) \\ y(t)=\frac{5}{8} c_1 e^t \sin (8 t)+\frac{1}{4} c_2 e^t (3 \sin (8 t)+4 \cos (8 t))$$
Now, what happens as $t$ approaches infinity?
You could:
Here is a phase portrait (what do you notice about the solutions):
Lastly, if you want to find the equilibrium points, you would simultaneously solve $x'=y'=0$ and have:
$$-5x-20y = 0,5x+7y = 0 \implies (x, y) = (0,0)$$
You can now find the Jacobian and evaluate the eigenvalues of it at that critical point.
The Jacobian of this is just:
$$J = \left( \begin{array}{cc} -5 & -20 \\ 5 & 7 \\ \end{array} \right)$$
of course, the eigenvalues are:
$$\lambda_{1,2} = 1~ \pm ~ 8i$$
Exactly the same as the way you did it (since this is a linear system, this is not very enlightening as we could have easily written this straight off from the coefficients of the linear system).