Difference between joint probability and mutual information on a Venn Diagram

867 Views Asked by At

We all know the usual Venn diagram for mutual information:

enter image description here

source Wikipedia.

The visualization and description of joint entropy H(X,Y) make it appear as though it is the same as mutual information I(X;Y), which of course it is not.

For example, we know that joint probability is the intersection of two events, and is where both circles overlap each other.

But mutual information is also where both circles overlap each other.

Wikipedia says the joint entropy is "the area contained by both circles." But this is also the mutual information.

Why is joint entropy and mutual information being depicted the same way?

2

There are 2 best solutions below

3
On

Wikipedia says the joint entropy is "the area contained by both circles." But this is also the mutual information.

No. The joint entropy is the area enclosed by the union of both circles. The mutual information is the intersection.

This alternative diagram (from MacKay's Information Theory, Inference, and Learning Algorithms) can help: enter image description here

0
On

This venn diagram from Thomas and Cover

enter image description here

is slightly different than the one you posted above in that it has arrows making it clearer that $H(X,Y)$ is apparently not the middle portion of the Venn Diagram and instead the union of X and Y as described in the answer by leonbloy. I agree this is confusing since I am used to seeing $p(X,Y)$ which is defined as the intersection.