It seems quite straightforward that $\frac1z-\frac{1}{z-1}$ does not have an antiderivative on $0<|z|<1$, because the singularity at $0$ makes integrals path-dependent. However, I still find it a bit tricky to prove this rigorously.
Let $f(z)$ be an analytic fuction on $0<|z|<1$ such that $$ f'(z)=\frac1z-\frac{1}{z-1}. $$ Let $z_0$ be a point satisfying $0<|z_0|<1$, and let $\gamma:[0,1]\to \mathbb C$ such that $\gamma(0)=z_0,\gamma(1)=z$. Then, $$ \int_\gamma f'(z)dz=\int_0^1 f'(\gamma(t))\gamma'(t)dt\\ =\int_0^1 \frac{d}{dt}(f(\gamma(t))) dt=f(\gamma(1))-f(\gamma(0))\\ =f(z)-f(z_0), $$ so if $z=z_0$ then $\int_\gamma f'(z)dz=0$. However, if $\gamma(t)=r e^{2 \pi it}, 0<r<1$, then by residue theorem, $\int_\gamma f'(z)dz=2 \pi i \text{ Res}(f',0)=2 \pi i\neq 0$, which is a contradiction.
Is that a perfect proof?
Your argumentation is fine. The first part can be generalized: a holomorphic function $f$ in $D$ has an antiderivative if and only if $$ \int_\gamma f(z) \, dz = 0 $$ for all closed (rectifiable) paths in $D$.
As you correctly demonstrated, that condition is not satisfied for $f(z) = \frac1z-\frac{1}{z-1}$ in the punctured unit disk.