The additive property of mutual information

Question

Let $X,Y_1,Y_2$ be three random variables. Let $I(X;Y)$ denote the mutual information of $X,Y$. My question: Does the inequality $I(X;Y_1,Y_2) \le I(X;Y_1)+I(X;Y_2) $ hold? Or, in what condition, does it hold?

Answer 1

$$I(X;Y_1,Y_2)=H(Y_1,Y_2)-H(Y_1,Y_2|X)=H(Y_1)+H(Y_2)-I(Y_1;Y2)-H(Y_1,Y_2|X)$$

$$I(X;Y_1)=H(Y_1)-H(Y_1|X)$$ $$I(X;Y_2)=H(Y_2)-H(Y_2|X)$$ Thus $$\begin{align} I(X;Y_1,Y_2)-\left(I(X;Y_1)+I(X;Y_2)\right)&=H(Y_1|X)+H(Y_2|X)-I(Y_1;Y2)-H(Y_1,Y_2|X)\\ &=I(Y_1;Y_2|X)-I(Y_1;Y_2) \end{align}$$ But $I(Y_1;Y_2|X)-I(Y_1;Y_2)$ which is called interaction information can be positive, negative, or zero.

See here and here for more information on positive and negative interaction information.

Answer 2

From the chain rule property of mutual information, it holds

$$ \begin{align} I(X;Y_1, Y_2)&=I(X;Y_1)+I(X;Y_2|Y_1)\\ &\leq I(X;Y_1)+I(X;Y_2), \end{align} $$

where the inequality holds only if

$$ I(X;Y_2|Y_1) \leq I(X;Y_2) \tag{1} $$

Note that $(1)$ does not hold in general (as also noted my @msm). One case where it does hold is when the variables form a Markov chain of the form $Y_1\rightarrow X \rightarrow Y_2$, i.e., it holds

$$ p(y_1,x,y_2)=p(y_1)p(x|y_1)p(y_2|x). $$