Continuity of Kullback-Leibler Divergence/Relative Entropy

130 Views Asked by At

Let $Y$ be a measurable subset of an Euclidean space $\mathbb{R}^n$. It is known that the Kullback–Leibler divergence $D_{KL}$ between probability measures on $Y$ is lower-semicontinuous but in general is not continuous with respect to the weak convergence of measures. However, all the examples I have found involve some failure of absolute continuity between the measures involved.

Therefore the following seems a natural question Is $D_{KL}$ continuous over the set of probability measures that are mutually absolutely continuous with respect to some $\mu^* \in \Delta(Y)$ where $\mu^*$ has full support on $Y$?

More specifically, it is true that if $(\nu_n,\mu_n)$ (weakly) converges to $(\nu,\mu)$ and $\nu_n,\mu_n,\nu,\mu$ are all absolutely continuous with respect to $\mu^*$ (for all n) then $\lim_n D_{KL}(\mu_n,\nu_n)$ exists and is equal to $D_{KL}(\mu,\nu)$? If yes, do you know a reference?