I'm working on information theory and I get confused by the information(venn) diagram. Is there any one help me with the following confusion?
As defined, mutual information $I(X;Y)$(Figure a) is described as the amount of common information shared by $X$ and $Y$ and conditional mutual information(CMI) $I(X;Y|Z)$(Figure b) can be interpretated as the amount of unknown common information given $Z$.
Following the above interpretation, I have the following intuitive idea: Since I already know some information(variable $Z$), I have the chance to learn about the common information shared by $X$ and $Y$. Hence I have less unkown information. The idea can be expressed by the following: $$ I(X;Y)\geq I(X;Y|Z) $$
However, I can't prove the above inequality. Can anyone tell me if there's any problem with the above idea? Thanks for any help.
The inequality is wrong; the typical counterxample is $Y=X+Z$ ,where $X$ and $Z$ are independent, taking values $\{0,1\}$ with equal probability and the sum is modulo two (XOR); here $I(X;Y)=0$ and $I(X;Y|Z)=1$. In general, the mutual information between two variables can increase or decrease when conditioning on a third variable.
The Venn diagrams applied to joint entropies are misleading when more than two variables are involved. In your case, it seems to imply that the center region (the triple intersection) should have a non-negative value, but that's false. See my related answer here.