Entropy Theory and Conditional Entropy

Question

Peter Walters An Introduction to Ergodic Theory.

Chapter 4

Entropy

Page 83 & 84

How did they duduce the last formula ?

Answer 1

Recall from properties of conditional expectation that

1. (Conditioning a.k.a. law of total expectation) For every $f\in L^1(X,\mathscr{B},m)$, we have \begin{align} \int f\,\mathrm{d}m &= \int \mathbb{E}[f\,|\,\mathscr{C}]\,\mathrm{d}m \;. \end{align}

2. (Pulling out known factors) For every $u,v:X\to\mathbb{R}$ such that $u,uv\in L^1(X,\mathscr{B},m)$ and $v$ is $\mathscr{C}$-measurable, we have \begin{align} \mathbb{E}[uv\,|\,\mathscr{C}] &= \mathbb{E}[u\,|\,\mathscr{C}] v \end{align} $m$-almost surely.

Now back to your case, for each $i$ we have \begin{align} \int \underbrace{\chi_{A_i}\log\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}]}_{f} \,\mathrm{d}m &= \int \mathbb{E}\Big[ \underbrace{\chi_{A_i}}_{u}\smash{\overbrace{\log\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}]}^{v}} \,\Big|\,\mathscr{C} \Big] \,\mathrm{d}m \\ &= \int \mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}] \log\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}] \,\mathrm{d}m \;, \end{align} from which can conclude the equality on the 3rd line of page 84.

To see the equality on the 2nd line of page 84, note that for each $j$ and $m$-almost every $x\in C_j$, the function $\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}](x)$ has the constant value $m(A_i\,|\,C_j)$. This is intuitively clear, but can formally be seen by verifying that the function \begin{align} g(x) &:= \sum_{j}\chi_{C_j}(x)m(A_i\,|\,C_j) \end{align} is a version of the conditional expectation $\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}]$ (i.e., it satisfies the definition). Thus, \begin{align} -\sum_{j=1}^k\int\chi_{A_i}\log\mathbb{E}[\chi_{A_i}\,|\,\mathscr{C}]\,\mathrm{d}m &= -\sum_{i=1}^k\int \sum_{j=1}^p\chi_{A_i}\chi_{C_j}\log m(A_i\,|\,C_j) \,\mathrm{d}m \\ &= -\sum_{i,j}\Big(\log m(A_i\,|\,C_j)\Big)\int\chi_{A_i\cap C_j}\,\mathrm{d}m \\ &= -\sum_{i,j}m(A_i\cap C_j)\log m(A_i\,|\,C_j) \;. \end{align}