I would like to determine the stability of equilibrium point $x=0$ of the differential equation $$\frac{dx}{dt}=x-\sqrt{x}, \\ x(t) > 0$$ By the standard testing procedure, I calculated $f'(x)$ as $1-\frac{1}{2}x^{\frac{1}{2}}$. Plugging $x=0$ into the $f'(x)$ gives $f'(0)=1>0$. Then we have $x=0$ is unstable.
However, when I plot the slope field of the DE.
I found that the trajectories are actually approaching $x=0$. Why?
You do not need to calculate $f'(t)$, you already have this expression given by the differential equation. Since, this system is time invariant, you just need to calculate $x-\sqrt{x}=0$, which has solutions at $1$ and $0$, as you can see from your plot.
Because $x-\sqrt x$ will be negative between $0$ and $1$, and positive afterwards, it follows that the trajectories will approach zero from above and diverge away from one. This is again confirmed by your plot.
You can plot $x-\sqrt x$ and see what values the derivative is going to take. If it is negative, the derivative will be negative and so the trajectories will be directed downward, it it is positive, they will be going upward. At the zeroes of this function, you will have the equilibrium points, and by looking at the plot's negative and positive values, you can tell the nature of the trajectories.