Does similarity of two probability distributions imply independence?

Question

I have two probability distributions and am using KL divergence to calculate the similarity between the two. If the divergence value is extremely small, I can say the two are similar in nature but can I also imply that they are two independent distributions?

Answer 1

No. It doesn't make sense to talk about the independence of two probability distributions. Instead you can talk about the independence of two random variables on the same probability space.

Here's an example. Let $\mu$ be the probability measure on $\{0,1\}$ so that $\mu(\{i\}) = 1/2$ for each $i \in \{0,1\}$. Then $D_{KL}(\mu\|\mu) = 0.$

Let $\Omega = \{0,1\}^2$, let $\mathcal{F} = 2^{\Omega}$ be the power set of $\Omega$ and let $\mathbb{P}$ be the uniform probability measure over $(\Omega,\mathcal{F})$. Let $X,Y:\Omega \to \{0,1\}$ be defined as follows:

$$X(\omega) = \omega_1 \text{ and } Y(\omega) = \omega_2.$$

Then $\mathbb{P}(X = 1) = \mathbb{P}(X=0) = \mathbb{P}(Y=1) = \mathbb{P}(Y=0) = 1/2$, so $X$ and $Y$ are both distributed like $\mu$, but they are independent.

We can then let $\mu_X = \mu$ and $\mu_Y = \mu$ be the respective probability distributions of $X$ and $Y$. Then $D_{KL}(\mu_X\|\mu_Y) = D_{KL}(\mu\|\mu) = 0$, but $X$ and $Y$ are independent.

At the same time, $D_{KL}(\mu_X\|\mu_X) = D_{KL}(\mu\|\mu) = 0$, but $X$ is certainly not independent from itself.

Thus, KL divergence says nothing about the independence or dependence of the two probability distributions in question.