$\sum p(x,y) \log p(x,y) = \sum p(x,y) \log p(x)p(y)$ (zero mutual information) implies independence

638 Views Asked by At

Let $(X,Y)$ be jointly distributed with probability mass function $p(x,y)$. What can be concluded if $\sum p(x,y) \log p(x,y) = \sum p(x,y) \log p(x)p(y)$? I thought about using Jensens inequality but that led to nothing.

Note that $\sum p(x,y) = 1$ are probabilities and we don't know wether $X$ and $Y$ are dependent or not.

1

There are 1 best solutions below

6
On BEST ANSWER

If the mutual information between $X$ and $Y$ is zero, then $X$ and $Y$ are independent. Here's a proof:

Define $$g(x,y):=\frac{p(x)p(y)}{p(x,y)}.$$ With this definition we have $$\sum p(x,y)g(x,y) = \sum p(x)p(y)=1,\tag1$$ where the sum is taken over all possible $(x,y)$ pairs. If the claimed equality holds, then $$ \begin{align} 0&=0 - \sum p(x,y)\log g(x,y)\\ &\stackrel{(1)}=\sum p(x,y)g(x,y) - 1 - \sum p(x,y)\log g(x,y)\\ &=\sum p(x,y)\left [g(x,y)-1 - \log g(x,y)\right]\tag2 \end{align} $$ Note that the bracketed quantity in (2) is nonnegative (since $\log t\le t-1$ for all $t$), so (2) is a sum of nonnegative terms. If the sum is in fact zero, then the bracketed quantity is identically zero: $$ \log g(x,y) = g(x,y) -1\quad\text{for all $x,y$} $$ But $t=1$ is the only value of $t$ for which $\log t=t-1$. This implies $$g(x,y)=1 \quad\text{for all $x,y$},$$ i.e., $X$ and $Y$ are independent.