Given equations $x^\prime=x(1-x-2y)$ and $y^\prime=y(1-y-4x)$ find all the singular solutions in the upper quadrant, $x\geq 0, y\geq 0$ and determine the type and stability.
So for singular solutions I believe I want points such that:
$x(1-x-2y)=y(1-y-4x)=0$
Which gives me $x=0,y=0$ or $(1-x-2y)=(1-y-4x)\implies y=3x$
So I have solutions $(0,0),(0,1),(1,0),(1/7,3/7)$
So I have the Jacobin matrix, $J=\begin{pmatrix} 1-2x-2y & -2x\\-4y&1-2y-4x\\ \end{pmatrix}$
And I need to evaluate the eigenvalues at different critical points.
$J(0,0)=\begin{pmatrix} 1&0\\0&1\\ \end{pmatrix}$
Which is already diagonal and has $1$ repeated real positive eigenvalue.
$J(0,1)$ has $1$ repeated negative eigenvalue
$J(1,0)$ has eigenvalues $-3,0$
$J(1/7,3/7)$ has one positive and one negative eigenvalue.
We want to simultaneously solve
$$x(1-x-2y) = 0 \\y (1-y-4x)=0$$
Clearly, $(x, y) = (0, 0)$.
When $x = 0$, we have $y(1 - y) = 0 \implies y = 0, y = 1$.
When $y = 0$, we have $x(1 - x) = 0 \implies x = 0, x = 1$.
Then we have
$$(1-x-2y) = 0 \\ (1-y-4x)=0$$
This gives us $x = \dfrac{1}{7}, y = \dfrac{3}{7}$.
So, our critical points are
$$(x, y) = (0, 0), (0, 1), (1, 0), \left(\dfrac{1}{7}, \dfrac{3}{7}\right)$$
Here is a contour plot that verifies this
Can you continue?
Hints:
Here is a phase portrait as a guide
Find the Jacobin matrix and evaluate the eigenvalues at each critical point while keeping track of any degenerate cases.
Update: For the Jacobian, I get
$$J(x, y) = \begin{pmatrix} -2 x-2 y+1 & -2 x \\ -4 y & -4 x-2 y+1 \\ \end{pmatrix}$$
My answer might also be helpful.