Name or intuition for this quantity similar to mutual information?

Question

The mutual information between random variables $X$ and $Y$ is defined (in terms of their joint PDF or PMF $p$) [Wikipedia]

$$ I(X; Y) = \mathbb{E}_{p(X, Y)}\left[ \log \frac{p(X, Y)}{p(X)p(Y)} \right]. $$

Equivalently we can write $$ I(X; Y) = \mathbb{E}_{p(X, Y)}\left[ \log {p(X, Y)} \right] - \mathbb{E}_{p(X, Y)}\left[\log {p(X)p(Y)} \right]. $$

When doing some calculations, I got the following quantity, which is the same but swapping the independent and joint distributions in the right-hand side. Is there a name for it, or an intuitive interpretation of it?

$$ F(X, Y) = \mathbb{E}_{p(X, Y)} \left[ \log p(X, Y) \right] - \mathbb{E}_{p(X)p(Y)} \left[ \log p(X, Y) \right] $$

Answer 1

Mutual information can be written as the Kullback-Leibler (KL) divergence between the joint distribution $p(X,Y)$ and the product distribution $p(X)p(Y)$, $$I(X;Y) = D_{KL}(p(X,Y)\Vert p(X)p(Y))$$

There is also a symmetrized version of KL divergence, sometimes called Jeffreys divergence or J-divergence, $D_J(p\Vert q) = D_{KL}(p\Vert q) + D_{KL}(q\Vert p)$ (see e.g. https://arxiv.org/pdf/1009.4004.pdf).

Your expression $F(X,Y)$ is the Jeffreys divergence between the joint distribution $p(X,Y)$ and product distribution $p(X)p(Y)$: \begin{align} D_J(p(X,Y)\Vert p(X)p(Y)) &= I(X;Y) + D_{KL}(p(X)p(Y)\Vert p(X,Y))\\ & = H(X) + H(Y) - H(X,Y)- H(X)-H(Y) - \mathbb{E}_{p(X)p(Y)}[\log p(X,Y)] \\ & = \mathbb{E}_{p(X,Y)}[\log p(X,Y)] - \mathbb{E}_{p(X)p(Y)}[\log p(X,Y)] \end{align}

The "reverse" KL term $D_{KL}(p(X)p(Y)\Vert p(X,Y))$ is sometimes called lautum information, and is described in the following:

• Palomar and Verdú, "Lautum Information", IEEE Trans on Info Theory, 2008.