Does total variation convergence with additional hypothesis entail relative entropy convergence?

48 Views Asked by At

Let $(\rho_t)_{t\in[0,\infty]}$ be a family of positive probability densities on $\mathbb R^d$, namely $\rho_t>0$ and $$ \int_{\mathbb R^d} \rho_t(x)\, dx =1 \;.$$ Suppose that: $$ \int_{\mathbb R^d}\rho_t(x)\,|\log\rho_s(x)|\,d x \,<\infty \quad\textrm{for every }t\geq0\textrm{ and }s=0,t,\infty \;;$$

$$\|\rho_t-\rho_0\|_{TV} \,:=\, \frac{1}{2}\,\int_{\mathbb R^d}|\rho_t(x)-\rho_0(x)|\,d x \,\to\,0 \quad\textrm{as } t\to0\;;$$

$$D_{KL}(\rho_t\|\rho_\infty) \,:=\, \int_{\mathbb R^d}\rho_t(x)\,\log\frac{\rho_t(x)}{\rho_\infty(x)}\,d x \,\to D_0\,\in[0,\infty)\quad\textrm{as } t\to0\;.$$

Can we conclude that:

$$ D_{KL}(\rho_0\|\rho_\infty)\,=\, D_0 \quad?$$

1

There are 1 best solutions below

4
On

No, I don't think so. For notational convenience, I will work with discrete measures; you can convolve them with square pulses to get something continuous. Consider $p_\infty[i] \propto \alpha^{-|i|}$

Then, the relative entropy $D(p || p_\infty)$ for $p$ reduces to:

$\sum_{i \in \mathbb{Z}} p_i \log p_i + |i| p_i \log \alpha - \log 2(1 - \frac{1}{\alpha})$

Now, let $p_{\frac{1}{n}} = \frac{1}{n}$ on $n$, $\frac{1}{n}$ on $-n$, and $0$ otherwise. Then, note that the relative entropy converges to $-\log 2(1 - \frac{1}{\alpha}) + \log \alpha$, but $D(p_0 || p_\infty) = -\log 2(1 - \frac{1}{\alpha})$