Consider the system of differential equations $$x'=\bigg[ \begin{matrix} a & 2 \\ a & 1 \end{matrix} \bigg] \cdot \bigg[ \begin{matrix} x_1 \\ x_2 \end{matrix} \bigg]$$ For what values of $a$ is the system asymptotic stable?
Investigate if the system is stable for $a=-1$ and for $a \geq 0$
My idea was to find the characteristic polynomial. $$P(\lambda)=\det(A-\lambda E) =(a-\lambda)(1-\lambda)-2a=\lambda^2-\lambda(1+a)-a$$
If the system has to be asymptotic stable, the solution must satisfy $x(t) \rightarrow0$ for $t \rightarrow 0$
We try to solve the characteristic equation: $$ \lambda^2-\lambda(1+a)-a =0 $$ and get $$\lambda=\frac{1}{2} +\frac{a}{2} + \frac{\sqrt{a^2+6a+1}}{2} \: \: \: \text{or} \: \: \: \lambda=\frac{1}{2} +\frac{a}{2} - \frac{\sqrt{a^2+6a+1}}{2}$$
But I'm not sure where to go from here. Can someone help me out?
The solution is asymptotically stable iff the real part of all the eigenvalues are negative (because that part leads to an exponential expression $e^{\mathrm{Re}(\lambda) t}$).
For two variables, this is equivalent to the trace of the matrix being negative and the determinant positive. You can see this by taking the different cases: $$\mathrm{Tr}(A)=\lambda+\mu\quad(=r+iv+r-iv=2r),\qquad \det(A)=\lambda\mu\quad(=r^2+v^2)$$
Thus, stable solutions occur when $$a+1<0,\qquad a-2a>0$$ $$\therefore\quad a<-1$$
Note: Your solution has an error in the characteristic polynomial (sign of last term).