convergence of entropy and sigma-fields

Question

This question is related to this one. Let $(X_1, X_2, \ldots)$ be a sequence of random variables such that each $X_n$ takes its values in a finite space, say $\{0,1\}$, and the $\sigma$-field $\sigma(X_1, X_2, \ldots)$ is discrete (generated by a denumerable partition of events). The maximal possible entropy of $(X_1, \ldots, X_n)$ is $n \log 2$. Is it possible that $\frac{H(X_1, \ldots, X_n)}{n}$ does not go to zero as $n\to \infty$ ?

Answer 1

No, this is impossible. Fix any $\epsilon>0$. Then, since, by assumption, the space is discrete, there is some finite set $\cal A$ of sequences $(X_1,X_2,\dots)$ such that ${\Bbb P}((X_i)\in {\cal A})>1-\epsilon$. Let $A$ be the event $(X_i)\in{\cal A}$. Then \begin{eqnarray*} H(X_1,\dots,X_n)&\le& H(X_1,\dots,X_n,A)\\ &=& H(A)+H(X_1,\ldots,X_n\mid A){\Bbb P}(A)+H(X_1,\ldots,X_n\mid A^C)(1-{\Bbb P}(A))\\ &\le& H(A)+H((X_i)\mid A)+n\epsilon\log 2. \end{eqnarray*} Since $\cal A$ is finite, $H((X_i)\mid A)$ is finite, so $$ \limsup_n \frac{H(X_1,\ldots,X_n)}{n}\le \epsilon \log 2, $$ and letting $\epsilon\to 0$, $$ \lim_n \frac{H(X_1,\ldots,X_n)}{n}=0. $$