Given the system $$x' = x(1-r)+y(x-r) \\ y' = y(1-r)+x(r-x)\\$$ where $r^2=x^2+y^2$. I don't know how to use the definition of stability of equilibrium to show that the equilibrium $(1,0)$ is unstable.
Definition: An equilibrium $\bf{\bar{x}}$ of an ODE is stable if for all $\epsilon > 0$, there exists $\delta >0$ such that $||\bf{x_0}-\bf{\bar{x}}||<\delta$ implies $||\bf{x(t)}-\bf{\bar{x}}||<\epsilon$ for all $t \geq 0$.
I have evaluated $$r(t)=\frac{r(0)e^t}{1-r(0)+r(0)e^t}$$ and $\\||\bf{x(t)}-\bf{\bar{x}}|| $$=\sqrt{(x(t)-1)^2+(y(t))^2}=\sqrt{(r(t))^2-2x(t)+1}$ and $||\bf{x_0}-\bf{\bar{x}}||$$=\sqrt{(r(0))^2-2x(0)+1}$. But I don't know the next step.
You can change the ODE from the cartesian coordinate to the polar cordinate:
$rr'=xx'+yy'$. Therefore, $rr'=(x^2+y^2)(1-r)=r^2(1-r)$, and so $r'=r(1-r)$.
We can conclude that:
For $0<r<1$, $r'>0$ and r is increasing.
For $r>1$, $r'<0$ and r is decreasing.
Hence, we approach the unit circle in this system, and it is wrong to say the equilibrium (1,0) is unstable.