Upper bound on the mutual information for a mixture distribution

Question

Assume we have two random variables $X$ and $Y$. Suppose further that $X$ is generated from a mixture distribution $P$ with $n$ components $P_i$ with corresponding weights $w_1,\dots,w_n$. Is it the case that the mutual information $I(X;Y)$ is bounded above by the weighted sum of the mutual information between $Y$ and each of the mixture components of $X$? That is, is it the case that $I(X,Y) = D_{KL}(P||Q)\leq \sum_{i}w_{i}D_{KL}(P_i||Q)$?

Answer 1

Let $Z$ be the indicator variable of the mixture. Assume $Z$ (but not $P_i$) is independent of $Y$.

Then $H(X) = H(X|Z) + H(Z) - H(Z|X)= \sum w_i H(P_i) + h(w) - H(Z|X)$

And $$I(X;Y)=H(X)-H(X|Y)=\sum w_i \, H(P_i) - H(Z|X) - \sum w_i\, H(P_i|Y) + H(Z|X,Y)\\ = \sum w_i \,I(P_i,Y) - I(Z,Y|X) \le \sum w_i \,I(P_i,Y)$$

Then, yes, the conjecture is true. But subject to the above (non trivial) assumption: that $Y$ is not related to the mixing.

Counter example, if the assumption does not hold: suppose the components don't overlap (hence $H(Z|X)=0$), and take $Y=Z$ (hence $H(Z|Y)=0$). Then

$$I(X;Y)=\sum w_i \,I(P_i,Y) +H(Z)$$