I am trying to find where there are stable of equilibrium points of this system such that $x,y \not= 0 $ in the region of the parameter space, with parameters $a$, $b$, and $c$ all being positive constants.
The system is: $$x' = x(1-x)-\frac{axy}{x+c}$$ $$y'= by(1-\frac{y}{x})$$
I am only concerned in the first quadrant of the system, namely where $x,y > 0$.
My initial thoughts were to find all the equilibrium points allowed by my conditions, linearize the system, then analyze the signs of the eigenvalues of the equilibrium points in the Jacobican matrix. I started by trying to identify all the equilibrium points. Since $y \not= 0$, it is clear that the any equilibrium points must be $x=y$. Yet, when I try to solve $x'=0$, it becomes very messy, and plugging in my values for $x$ back into the Jacobian matrix would make it even worse.
It started to become clear to me that I must not be doing something correctly, or that I am missing something important. Is there a better and more practice way to find where in $abc$-space I get stable equilibrium points? Maybe by finding a Liapunov function?
As soon as you have $y=x$, any equilibrium satisfying your requirements is a positive solution of $(1-x)(x+c)-ax=0$, that is the quadratic equation $x^2+(a+c-1)x-c=0$. Since $c>0$, it follows that you have one positive root. You can write this root in terms of $a$ and $c$. Then you compute the Jacobian of your system, $$J = \begin{bmatrix} \frac{a\,x\,y}{{\left(c+x\right)}^2}-\frac{a\,y}{c+x}-2\,x+1 & -\frac{a\,x}{c+x}\\ \frac{b\,y^2}{x^2} & \frac{b\,\left(x-2\,y\right)}{x} \end{bmatrix},$$ and substitute the obtained solution as $x$ and $y$. You obtain the linearization $2\times 2$ matrix where all elements are functions of $a$, $b$, and $c$. Then you compute its deterministic polynomial and check when all coefficients are positive. That gives you the desired stability condition. It looks a bit messy to put the equations here, but a CAS should help.