Which one is bigger $D(P\Vert Q)$ or $D(Q \Vert P)$?

150 Views Asked by At

In general the Kullback-Leibler divergence is asymmetric. If $P$ and $Q$ are two distributions $D(P\Vert Q) \ne D(Q\Vert P)$. However, I was wondering if there are situations where we can say which one is bigger $D(P \Vert Q)$ or $D(Q \Vert P)$?

In particular if $P(x,y)$ is a joint distribution with marginals $P_X(x)$ and $P_Y(y)$ and we set $Q(x,y) = P_X(x) \times P_Y(y)$ can we compare $D(P \Vert Q)$ and $D(Q \Vert P)$?

1

There are 1 best solutions below

0
On BEST ANSWER

The particular case with $P=P_{X,Y}$ being the joint distribution with marginals $P_X$ and $P_Y$, and $Q=P_X\times P_Y$ is addressed in the paper

https://www.princeton.edu/~verdu/lautum.info.pdf

$D(P_X \times P_Y || P_{Y,X})$ is called the lautum information. In the paper, it is found that for the BSC (Theorem 12) and Gaussian channel (Theorem 15), $L(X;Y) \geq I(X;Y)$.