Say I have the following first-order system for $n$:
$$ \dot{n} = \frac{Gnp}{Gn+f} - kn, $$
where $G$, $f$, and $k$ are always positive, but $p$ can take either sign.
First, I want to find the critical points $n^*$ of this system. One is obviously $n^*=0$. I can solve for the other one to get $n^*=\frac{pG-fk}{Gk}$. Further algebraic rearrangements reveal that the second critical point is negative if $p<\frac{fk}{G}=p_c$ and positive if $p > p_c$.
What should I do to show that $n^*=0$ becomes unstable for $p>p_c$?
You can check the stability of a fixed point $n^{*}$ of the system $\dot{n} = f(n)$ by checking its derivative at $n^{*}$.
If $f^{'}(n^{*}) < 0$, then $n^*$ is a stable fixed point. If $f^{'}(n^{*}) > 0$, the fixed point will be unstable. For $f^{'}(n^{*}) = 0$, we cannot tell whether the fixed point is stable or unstable and have to get another case.
Here, using the quotient rule, we can calculate $f^{'}(n) = \frac{1}{(Gn + f)^{2}} [(Gn + f)Gp - (Gnp)G]-k$. For unstabilty check for two fixed points ,
$case-1 :$
when $$ n^{*} = \frac{pG - fk}{Gk}$$
$f^{'}(n) = \frac{Gp}{(Gn + f)} - \frac{G^{2}np}{(Gn + f)^2} - k > 0$ So we have three terms above , if we workout for first term it comes out as $k$ and this cancels out with the $-k$ (the third term) , so we are only left with $-\frac{G^{2}np}{(Gn + f)^{2}} > 0$ , and when we substitute in the latter ones $n = \frac{pG - fk}{Gk}$ we arrive at $-k (1- \frac{fk}{pG}) > 0$ leading to $p < \frac{fk}{G}$ ,so when $p < \frac{fk}{G}$, the fixed point is unstable .
$Case-2$
when $$n^{*} = 0$$
And now for $n^{*} = 0$ , substituting in the expression of $f^{'}$ above we get $\frac{Gp}{f} - k > 0$ implies for $p > (p_{c} = \frac{kf}{G})$ , $n^{*} = 0$ is an unstable fixed point.
Hope this helps !