Let $\alpha\mapsto\varepsilon_\alpha$ be the enumeration of the $\varepsilon$-numbers--that is, those $\alpha$ such that $\omega^\alpha=\alpha$--by the ordinals.
If we know that countable unions of countable sets are countable, (slightly weaker than countable choice), then we can show fairly easily that $\varepsilon_\alpha$ is countable iff $\alpha$ is countable.
I think I saw a result at some point that $\varepsilon_0$ is countable without relying on any choice principle (though I can't recall where I saw that). Is this true? More generally, for which $\alpha$ can we conclude that $\varepsilon_\alpha$ is countable in ZF?
Update: I have been able to prove (in ZF) that if $\varepsilon_\alpha$ is countable, then $|\alpha|<\aleph_1$, so that's one direction. I'm still a bit uncertain about the answers I've got so far.
Indeed you do not need the axiom of choice to prove that $\epsilon_\alpha$ is countable for every countable ordinal $\alpha$.
To see this, observe first that the meaning of $\epsilon_\alpha$ is the same in the set-theoretic universe $V$ as it is in $L$, where ZFC holds. Furthermore, in ZFC an elementary proof by transfinite induction shows that the cardinality of the ordinal $\epsilon_\alpha$ is precisely $\max\{|\alpha|,\omega\}=|\alpha+\omega|$. Your observation about $\epsilon_0$ is simply the first case of this, with the later cases not really any more difficult. Combining these two observations, in $V$ with only ZF we get a bijection between $\epsilon_\alpha$ and $\alpha+\omega$, and so if $\alpha$ is countable in $V$---whether or not it is countable in $L$---then so is $\alpha+\omega$ and we conclude that $\epsilon_\alpha$ is countable in $V$, as desired.
To respond to your comment below, let me describe a direct argument showing that $\omega^\alpha$ is countable whenever $\alpha$ is, without using the axiom of choice and without appealing to the constructible universe $L$.
One may proceed as follows. The ordinal $\omega^\alpha$ is precisely the order type of the finite support functions from $\alpha$ to $\omega$, where finite support means that only finitely many values are non-zero, ordered lexically as in our manner of ordering numbers by their digits. This is like a base-$\omega$ representation of the ordinals, using representations with $\alpha$ many digits (but only finitely many non-zero digits). That $\omega^\alpha$ is the order type as described can be proved by transfinite induction. Now, the point is that without AC we can prove that there are only countably many finite functions from a countable set to $\omega$, and thus $\omega^\alpha$ is countable whenever $\alpha$ is.
I believe that one may also develop such a concrete representation of $\epsilon_\alpha$, and use this concrete representation to prove that $\epsilon_\alpha$ is countable whenever $\alpha$ is, but I will leave this task for someone else, who will surely post it before long...