What are the most general hypotheses on the domain under which I can apply Rouche's theorem? I have seen some books state that you need the domain of the functions to be simply connected. Is this true? Can I use Rouche's theorem for more general domains?
For example, say I have two functions f and g which are holomorphic on an open set containing an annulus. If $|f|>|g|$ on the boundary of the annulus, can I say that $f$ and $f + g$ have the same number of zeros inside the annulus?
I have searched all over, and I have seen other posts about Rouche's theorem on this website, but none of them seem to be completely precise about their hypotheses. And help or references you can provide would be wonderful.
What you need is a domain with well-behaved boundary. Rouché's theorem says that two functions $f,g$ holomorphic on a neighbourhood of $\overline{V}$, where $V\subset \mathbb{C}$ is open, have the same number of zeros in $V$, if $\lvert f-g\rvert < \lvert f\rvert + \lvert g\rvert$ on $\partial V$ and the boundary of $V$ can be traced by finitely many paths of integration, such that $V$ lies to the left of each of these paths.
Such open sets are in German called "positiv beranded", I don't know if there is an English term for such open sets.
The requirements on $\partial V$ ensure that integrating over $\partial \Omega$ is well-defined, and that we can prove the residue theorem (Cauchy integral theorem, integral formula) for integrals over $\partial V$ by approximating $\partial V$ by piecewise smooth cycles from inside $V$.
Annuli are totally fine, every domain with smooth boundary works. One has two boundary components for annuli of course.
The proof just notes that
$$N(\lambda) := \frac{1}{2\pi i} \int_{\partial V} \frac{(1-\lambda)f'(z) + \lambda g'(z)}{(1-\lambda)f(z) + \lambda g(z)}\,dz$$
is constant on $[0,1]$ since $(1-\lambda)f + \lambda g$ doesn't vanish on $\partial V$ for $\lambda \in [0,1]$.
One can generalise, it is not necessary that the functions are holomorphic in a neighbourhood of $\overline{V}$, it is sufficient that the functions are holomorphic on $V$ and they together with their first derivatives extend continuously to $\overline{V}$.