I know that there already is a similar question out there but I am not really satisfied with that answer. I think it is too complicated. Assume that $G \subseteq \mathbb{C}$ is open and simply connected, $P \subseteq \mathbb{C}$ has finite cardinality and $f: G \setminus P \rightarrow \mathbb{C}$ is holomorphic. Then
$f$ has an antiderivative on its domain $\iff $ $\mathrm{Res}_p(f) = 0 ~\forall p \in P$.
Can I argue the following way? The crucial part is the second implication.
"$ \implies$": Since $f$ has an antiderivative, all integrals over closed rectifiable curves vanish. If I let $p \in P$ be arbitrary and $\varepsilon > 0$ so small that $\partial B_\varepsilon(p) \subseteq G \setminus P$, then according to the residue theorem: $$ 0 = \int_{\partial B_\varepsilon(p)} f(z)~\mathrm{d}z = 2 \pi i\mathrm{Res}_p(f) $$
"$\impliedby$": For any closed rectifiable curve $\gamma$ we have: $$ \int_\gamma f(z)~\mathrm{d}z = \sum_{p \in P} I_p(\gamma) \mathrm{Res}_p(f) = 0 $$ Therefore the existence of a primitive.
Under the given conditions on $G$, $P$, and $f$, the following three statements are equivalent:
(a) $f$ has an antiderivative in $G$.
(b) $\operatorname{Res}_p(f) = 0 $ for all $ p \in P$.
(c) $\int_\gamma f(z) \, dz = 0$ for all closed rectifiable curves $\gamma$ in $G$.
What you have demonstrated is that $(a) \implies (b)$ and that $(b) \implies (c)$.
It remains to show that $(c) \implies (a)$, and that can be done by showing that $F(z) = \int_{z_0}^z f(w) \, dw$ (which is well-defined under the condition (c)) satisfies $F' = f$.