Mutual information of discrete RVs which converge in distribution to a continuous RV

183 Views Asked by At

$\mu_{X_n,Y_n}$ is a sequence of discrete joint-distributions on $\mathbf{R}^2$ that converge weakly to a continuous measure $\mu_{X,Y}$. That is, for any continuous function $f:\mathbf{R}^2\rightarrow \mathbf{R},$ then $|\int f\ d\mu_{X_n,Y_n}-\int f\ d\mu|\rightarrow 0$.

Is it true that $I(X_n,Y_n)\rightarrow I(X,Y)$?

I have been trying to prove this by realizing that the sequence $I(X_n,Y_n)$ is kind of like a monte-carlo estimation of the integral $I(X,Y)$

2

There are 2 best solutions below

1
On BEST ANSWER

Let n be a large even number. Take n equally spaced points along the unit interval. A probability distribution which gives equal weight to all these points converges weakly to the uniform distribution. The product of two copies of this distribution converges weakly to uniform on the unit square. Since this distribution is equal to the product of it's marginals, it has mutual information equal to 0.

Now take the same set of points, and this time discard half of them in a chequerboard pattern. Share out the probability equally between the rest. This also converges weakly to uniform for large n. However,mutual information is now the expectation of a random variable equal to constant the logarithm of two.

2
On

Too much characters for a comment. So, I'll leave it here...

$$I(X_n,Y_n)=\sum p_{ij}\ln(p_{ij})-\sum p_i\ln(p_i)-\sum p_j\ln(p_j)$$

where $p_{ij}=P_{X_nY_n}(i,j), p_i=P_{X_n}(i)$ and $p_j=P_{Y_n}(j)$.

For example, rewriting the entropy of $X_n$ (the second term) you have (using Taylor expansion of the logarithm)

$$\sum p_i\ln(p_i)=\sum(p_i-f(x_i)\triangle)\ln(p_i)+\sum f(x_i)\triangle\ln(f(x_i))+\sum f(x_i)\triangle\ln(\triangle)+\sum [(p_i-f(x_i)\triangle)+f(x_i)\triangle\cdot O((p_i-f(x_i)\triangle)^2]$$

where $f$ is the PDF of $X$.

So, if you can show that the first and the last terms converge to zero, the second term is the Riemann integral with $\triangle\rightarrow 0$ so that

$$\sum f(x_i)\triangle\ln(f(x_i))\rightarrow \int f(x)\ln(f(x))dx \text{ or } \sum p_i\ln(p_i) \rightarrow -H(X)$$