If we find the equilibrium point and linearize the system
$x'=-x+ay+x^2y\\ y'=b-ay-x^2y,$
we get that the point is $(b, \frac{b}{a+b^2})$ and the matrix associated with the linearilized system is
\begin{bmatrix} \frac{-a+b^2}{a+b^2} & a+b^2\\ -\frac{2b^2}{a+b^2} & -(a+b^2) \end{bmatrix}
To determine the nature of the equilibrium point we need to find the eigenvalues of this matrix. Finding the eigenvalues, we get this:
$\lambda = -\frac{\pm\sqrt{(a+b^2)^2[(a+b^2)^2+2(a-b^2)-4(a+b^2)]+(a-b^2)^2}+(a+b^2)^2+(a-b^2)}{2(a+b^2)}$.
From my understanding, since this is a biological model (Sel'kov model), a model for breaking down of sugar, $a$, $b >0$.
But I have no idea how to analyze $\lambda$ since I don't know what values $a$ and $b$ can take and how $a$ and $b$ are related. I'd appreciate some help please.