Entropy of an infinite sequence?

Question

Does an infinite sequence always have finite entropy? For example, doesn't $a_n=n$, the sequence of non-negative integers, have very low entropy? It feels like all "well-defined" sequences ought to have low entropy because there's very little surprise about the next element.

Answer 1

Your question is a little vague.

In the context of the Shannon entropy, one natural and usual measure of the "rate of information" of an infinite sequence (more aptly: of a discrete time stochastic process) is the entropy rate:

$$H_r = \lim_{n\to \infty} \frac{H(X_1,X_2 , \cdots H_n)}{n}$$

... if the limit exists, of course. Notice also that this is not the information of a single full sequence, but a normalized expected value.

Typically, $0< H_r < \infty$, and and in that case the entropy of the infinite sequence is infinite. In particular, if the sequence is produced by a stationary memoryless source (independent symbols with stationary distribution) then $H(X_1,X_2 , \cdots H_n)=n H(X_1)$ and $H_r = H(X_1)$

A little more general: for a stationary first order Markov process, $H_r = H(X_2 \mid X_1)$

If each symbol is totally predictable from the previous one, then $H_r=0$.

In your case, your sequence is not only predictable but also deterministic, hence $H_r=0$

This is not the end of the story, though. Because the Shannon entropy requires a probabilistic setting, and sometimes that does not seem very adequate. The typical example: which is the entropy rate of $X_n=$ digits of the decimal expansion of $\pi$? For an alternative approach to defining the "average information" of a sequence (and hence some alternative "entropy"), using an operational (computational) instead of probabilistic setting, you might look into Kolmogorov complexity

Answer 2

I think it makes sense if the terms of your series are all positive and the series converges. For example if $A=\sum_{n=1}^\infty a_n$, then you can define the probability distribution $p_n=a_n/A$. Hence, using the entropy definition $S=-\sum_{n=1}^\infty p_n\ln p_n$, which is the standard definition of statistical mechanics divided by the Boltzmann constant $k_B$. For example, we could define then the entropy of Riemann's zeta function $\zeta(s)=\sum_{n=1}^\infty\frac{1}{n^s}$. Using the probability distribution $p_n(s)=1/(\zeta(s)n^s)$, we obtain the "entropy" of the zeta function: \begin{align} S(s) &= \frac{1}{\zeta(s)}\sum_{n=1}^\infty\frac{1}{n^s}\ln(\zeta(s)n^s)=\ln\zeta(s)+\frac{s}{\zeta(s)}\sum_{n=1}^\infty\frac{\ln n}{n^s}=\ln\zeta(s)-\frac{s}{\zeta(s)}\zeta'(s)\\ &=\ln\zeta(s)-s\frac{d\ln\zeta(s)}{ds} \end{align} This is called the zeta distribution. See here: https://en.wikipedia.org/wiki/Zeta_distribution Another example is the Poisson distribution, which you can obtain from the expansion of $e^{\lambda t}$. The probability distribution you get from this is $p_n=\frac{e^{-\lambda t}(\lambda t)^n}{n!}$.