Given the nonlinear system,
$$ x' = xy = f(x,y)$$ $$ y' = x(1-x) = g(x,y) $$
The two points where $f=g=0$ are $(0,y)$ and $(1,0)$. I am confused about the theory on how one would continue the stability analysis with the 'point' $(0,y)$ even though it's not really a point. By looking at the phase portrait, the flow tends away from $y$ axes, indicating that it's possibly unstable? I've tried to look for material on this but I'm not entirely sure how to phrase what I'm searching for properly. Thanks!
$E=\{(0,y),\; y\in \mathbb R\}$ is a set of equilibrium points. The fact that there are infinitely many equilibrium points and they are not isolated does not prevent us from applying known stability theory results to them. For example, one can apply the Chetaev instability theorem to prove the instability of equilibrium points from the subset $E_+=\{(0,y),\; y>0\}$. Consider the function $$ V(x,y)=x. $$ For any point $(0,y_0)$ from $E_+$ it satisfies the conditions of the theorem, thus, the equilibrium points from $E_+$ are unstable.
For the system under consideration, however, this is not required, since it admits the first integral $(x-1)^2+y^2$. If we take into account that the direction of movement along the circles i.e. along the level lines $(x-1)^2+y^2=C$ is clockwise in the right half-plane and counterclockwise in the left half-plane, then the behavior of the system becomes clear: