Say we have a variation of the Lotka Volterra system:
$\frac{dx}{dt} = ax-bxy-cx^2$
$\frac{dy}{dt} = ny+mxy-py^2$
where $a,b,c,m,n,p$ are positive constants and $x_0$ and $y_0$ represent the initial population. The values of $a,b,c,m,n,p,x_0,y_0$ don't matter as we can experiment with different ones.
I'm having trouble finding the equilibrium points and their stability. Any help on this is appreciated.
To find the critical points, we want $x' = 0, y' = 0$, so have:
$$ \dfrac{dx}{dt} = ax-bxy-cx^2 = 0 \\ \dfrac{dy}{dt} = ny+mxy-py^2 = 0 $$
We can write this as:
$$x(a-cx -by) = 0 \\ y(n + mx -py) = 0 $$
It is clear that we satisfy these equations with $x = 0, y = 0$.
If we choose $x = 0$ in the first equation, the second gives us:
$$ y = \dfrac{n}{p}$$
If we choose $y = 0$ in the second equation, the first gives us:
$$ x = \dfrac{a}{c}$$
Lastly, we are left analyzing the cases:
$$a-cx -by = 0 \\ n + mx -py = 0 $$
Using your favorite method for a $2x2$ (RREF), this simultaneously gives:
$$ x = \dfrac{a p-b n}{b m+c p}, y = \dfrac{a m+c n}{b m+c p}$$
We are left with four critical points to investigate as:
$$(x, y) = (0, 0), \left(0 , \dfrac{n}{p}\right), \left(\dfrac{a}{c} , 0 \right), \left(\dfrac{a p-b n}{b m+c p}, \dfrac{a m+c n}{b m+c p}\right)$$
It is obvious that some of the constants cannot be zero for obvious reasons, for example, $p \ne 0$ in the second critical point.
You can test these critical points in the original system to verify they simultaneously produce $x ' = 0, y' = 0$.
Next, start analyzing these points for stability (your turn).