Estimating entropy rates conditioned on typical sets

42 Views Asked by At

Consider an ergodic process $Y_1^n$ that depends on an i.i.d. process $X_1^n$. We can estimate the entropy rate $\overline{H}(Y) = \lim_{n \rightarrow \infty} (1/n)H(Y_1^n)$ using the Shannon-McMillan-Breiman theorem, which says $g_n = -(1/n) \log P(Y_1^n) \rightarrow \overline{H}(Y)$. Instead, consider estimating this entropy rate using the probability $Q(Y_1^n) = \mathbb{P}(Y_1^n|X_1^n \in A_{\epsilon}^{(n)})$, where $A_{\epsilon}^{(n)}$ is the $\epsilon$-typical set of $X_1^n$ with $\mathbb{P}(X_1^n \in A_{\epsilon}^{(n)}) > 1-\epsilon$. That is, we want $\tilde{g}_n = -(1/n)\log Q(X_1^n)$ to be asymptotically close to $g_n$ with high probability. How can we prove this with minimal assumptions about $Y_1^n$? It is likely useful to bound the divergence $D(P||Q)$, but I'm not sure how to do this. Any help would be greatly appreciated.