Question about summation Chain Rule of Entropy and simplification of summation in the proof

20 Views Asked by At

So I was going through the proof on page 23 of https://www.icg.isy.liu.se/courses/infotheory/lect1.pdf, and I'm not quite understanding how the double summation simplifies to a single summation . The part where it goes

\begin{align} H(X, Y) &= -\sum_{x} \sum_{y} p(x,y) \log p(x) - \sum_{x} \sum_{y} p(x,y) \log p(y|x) \\ &= \sum_{x} p(x) \log (p(y) - \sum_{x} \sum_{y} p(x,y) \log p(y|x) \end{align}

If I assume independence between $x$ and $y$ then the double summation $-\sum_{x} \sum_{y} p(x,y) \log p(x)$ would simply as follows

\begin{align} -\sum_{x} \sum_{y} p(x,y) \log p(x) &= -\sum_{x} \sum_{y} p(x) p(y) \log p(x) \\ &= -\sum_{x} p(x)\log p(x) \sum_{y} p(y) \\ &= -\sum_{x} p(x)\log p(x) * 1 \\ &= H(X) \\ \end{align}

but how would I show this without assuming independence? Is there a trick to simplifying $-\sum_{x} \sum_{y} p(x,y) \log p(x) $ ?