Kullback-Leibler Approximation of Unbounded by Bounded Densities

54 Views Asked by At

Let $P$ denote a distribution on $[0,1]$ with density $p$, and let $KL(p \| q)$ denote the Kullback-Leiber divergence $$ KL(p\|q) = \int_0^1 p \log \frac{p}{q} dx. $$ Let $\mathcal C_{>0}([0,1])$ denote the collection of densities $q(x)$ such that $q(x)$ is bounded away from 0, i.e., $\inf_x q(x) > 0$ and continuous on all of $[0,1]$ (hence also bounded).

Question: Under what conditions does there exist a sequence $q_1, q_2, \ldots$ from $\mathcal C_{>0}([0,1])$ such that $KL(p \| q_n) \to 0$ as $n \to \infty$?

Partial progress: If $p(x)$ is bounded above and continuous on $(0,1)$, then the problem is trivial: just take $q_n(x) \propto p(x) + \delta_n$ for any sequence $\delta_n \downarrow 0$. Then $KL(p \| q_n) = \log(1 + \delta_n) + \int p_0 \log \frac{p_0}{p_0 + \delta_n} \le \log(1 + \delta_n) \to 0$.

I can also make progress when $p(x)$ is continuous on $(0,1)$, bounded away from $0$, but unbounded at the endpoints using some simple constructions. For example, suppose $p(x)$ is unbounded near $0$ and let $q(x) \propto p(x)$ on some interval $I_\delta = (\delta, 1)$ and $q(x) \propto p(\delta)$ otherwise. The normalizing constant is given by $\delta p(\delta) + P I_\delta$, which converges to $1$ provided that $\delta p(\delta) \to 0$. If we add the additional condition that $P |\log p| < \infty$ then $$ \lim_{\delta \to 0} KL(p \| q_\delta) = \lim_{\delta \to 0} \int_0^{\delta} p(x) \log \frac{p(x)}{p(\delta)} = \lim_{\delta \to 0} \int_0^\delta p(x) \log p(x) - P_0([0,\delta]) \log p(\delta) = 0. $$ The conditions $P |\log p| < \infty$ and $\delta p(\delta) \to 0$ (e.g., $\delta p(\delta) \to 0$ must occur if $p(x)$ is monotone on some neighborhood of the endpoints, and possibly I can dispense with this condition by picking $\delta$ carefully) seem to cover just about any density I would encounter in practice. Dispensing with $P|\log p| < \infty$ would be desirable, however.