I want to determine the stability of the equilibrium points of the equation $\dot{x}=x(x-1)$.
It is not difficult to see by drawing the phase line that $x=0$ is a sink and that $x=1$ is a source.
Intuitively this tells me that the solution $x=0$ is stable and that $x=1$ is unstable, but I wonder if there is a way to test it using Lyapunov's definition of stability.
Although you can use the definition of stability, this problem is better solved using Lyapunov functions.
Consider the equilibrium point $x_0=0$. The function $V_0(x)=x^2$ is a corresponding Lyapunov function as it is positive definite, continuously differentiable and its derivative along the trajectories $$ \dot V_0= 2x\dot x= 2x^2(x-1) $$ is negative in some deleted neighborhood of $x_0$. Hence, $x_0$ is an asymptotically stable equilibrium point.
Similarly, the function $V_1(x)=(x-1)^2$ is a Lyapunov function for the equilibrium point $x_1=1$. Its derivative along the trajectories $$ \dot V_1= 2(x-1)\dot x= 2x(x-1)^2 $$ is positive in some deleted neighborhood of $x_1$, thus, $x_1$ is unstable.