Transformation of mutual information to probability distribution

Question

Given the upper bound for mutual information of random variables $X$ and $Y$, $I(X;Y)\leq L$, what can we say about their joint distribution? I mean for example if $L=0$, then we know $p_{XY}(A\cap B)=p_X(A)p_Y(B)$ for $A$ and $B$ measurable sets, can we somehow transform $I(X;Y)\leq L$ into a property of $p_{XY}$. I am particularly thinking about bounding a quantity like $$\sup\{|p_{XY}(A\cap B)-p_X(A)p_Y(B)|\}$$ as a function of $L$.

Answer 1

There is an inequality called Pinsker inequality, expressing a relation between total variation and Kullback-Leibler divergence. For two probability distribution $Q$ and $P$ over the probability space $(S,\Omega)$, we have:

$$ \displaystyle\sup_{A\in\Omega}\{P(A)-Q(A)\}\leq\sqrt{\frac{1}{2}D(P||Q)} $$

We know $I(X;Y)=D\left(P(X,Y)||P(X)P(Y)\right)$. Therefore if $I(X;Y)<L$ then according to Pinsker inequality we have:

$$ \displaystyle\sup_{A\in\Omega}\{P(X,Y)-P(X)P(Y)\}\leq\sqrt{\frac{1}{2}L} $$