Consider the IVP for the $2$-dimensional dynamical system ($X=[0, \infty )^2$) $$\dot{x_1}=a-x_1-\frac{4x_1x_2}{1+x_1^2}$$ $$\dot{x_2}=bx_1 \bigg( 1- \frac{x_2}{1+x_1^2} \bigg)$$ for all $t \in I$, and $\textbf x (0)=(x_{10}, x_{20}) \in [0, \infty )^2$, with $a,b >0$.
I found that there is one fixed point at $$\bigg( \frac a5, 1 + \frac{a^2}{25} \bigg)$$ which I found to be hyperbolic. The eigenvalues of this were $$\lambda_{\pm} = \frac{3 a^2 \pm \sqrt{(-3 a^2+5 a b+125)^2-4 (10 a^3 b+25 a^3+625 a)}-5 a b-125)}{2 (a^2+25)}$$
The question is:
For such a hyperbolic fixed point $\textbf x ^*$ to the IVP, determine conditions on $a$ and $b$ so that the fixed point $\textbf x ^*$ is unstable (here, for hyperbolic fixed points, this is equivalent to the dimension of $W_{loc}^u (\textbf x ^*)$ being greater than $0$)
What am I meant to do in this? Since they want it to be unstable, then that means it can't a saddle point but has to be a spiral right? So the chunk inside the sqrt should be negative. And the real parts should all be positive for an unstable spiral.
A fixed point of a planar system is stable if and only if both eigenvalues of the Jacobian of the system, evaluated at that point, have negative real part. So, the fixed point is unstable if at least one of the eigenvalues has positive real part.
For the eigenvalues of a $2\times 2$ matrix, there is a nice characterization of the sign of the real part of the eigenvalues in terms of the trace and determinant of the matrix. This is because the characteristic polynomial of such a matrix $A$ takes the form \begin{equation} \lambda^2 - \text{tr }A\,\lambda + \text{det }A. \end{equation} Everything you need to know about the eigenvalues, being roots of this characteristic equation, is given in this picture:
So, you see that the fixed point is unstable if $\text{det }A < 0$, or if $\text{tr }A > 0$. In your case, we have \begin{align} \text{tr }A &= \frac{3 a^2 -5 a b - 125}{25 + a^2},\\ \text{det }A &= \frac{25 a b}{25 + a^2}. \end{align} Since it is given that both $a$ and $b$ are positive, $\text{det }A > 0$. That means that the only way the fixed point can be unstable, is when \begin{equation} \text{tr }A = \frac{3 a^2 -5 a b - 125}{25 + a^2} < 0. \end{equation} I'm sure you will have no problems obtaining conditions on $a$ and $b$ from the above inequality.