I have found various sources on the internet that say that power series are infinitely differentiable on their interval of convergence:
Once a function $f(x)$ is given as a power series as above, it is differentiable on the interior of the domain of convergence.
[...] power series are (infinitely) differentiable on their intervals of convergence [...]
But isn't every power series (infinitely) differentiable everywhere?
After all, a power series is just an infinite polynomial and a polynomial of degree $n$ is differentiable $n+1$ times. Source
Doesn't this imply, that a polynomial of "degree $\infty$" is differentiable $\infty$ times?
I think this was an issue of language and my understanding of what it means for a derivative to exist.
It is of course possible to "derive" (e.g. blindly following the rules of differentiation) term-by-term like this: $$ f(x) = \sum_{n=0}^\infty a_n(x-c)^n = a_0 + a_1(x-c) + a_2(x-c)^2 + a_3(x-c)^3 + ... $$ $$ f'(x) = \sum_{n=1}^\infty na_n(x-c)^{n-1} = a_1 + 2a_2(x-c) + 3a_3(x-c)^2 + ... $$
However, an infinite sum only exists, if it converges!
Therefore, if the infinite sum does not converge (which it might, depending on $x$), it does not exist, and therefore the derivative itself does not exist for those values of $x$.
The radius of convergence of the power series and its derivatives is the same: Source 1, Source 2.
Edit
I would like to highlight this statement by Kavi Rama Murthy from the comments, because it complements this answer: