Does bounded KL-divergence imply tightness?

Question

We have a sequence of probability measures $P_n, n≥1$, and a probability measure $Q$ such that $KL(P_n || Q) < v$ for each $n$, where $v$ is a finite constant and $KL(P||Q)$ is the Kullback–Leibler divergence, i.e. $KL(P||Q) ≔ ∫\ln\frac{dP}{dQ}(x)P(dx)$ whenever $P ≪ Q$ and the integral is finite.

We can assume that all probability measures are on the real line with the usual Borel σ-algebra (I am interested in the case of Polish spaces, but I suspect it does not make a difference here).

Does it follow that $P_n, n≥1$ is (uniformly) tight? That is, for each $ε > 0$ there is a compact $K$ such that $P_n[K^c] < ε$ for each $n$.

Answer 1

Since $P << Q$ By Radon Nikodyn Theorem: $P(x) = f(x) Q(x)$.

If we assume $\sup_x f(x) < \infty$ then : $$P(K^c)=\int_{K^c} P(x) dx = \int_{K^c} f(x) Q(x) dx \leq sup_x f(x) \times \int_{K^c} Q(x) dx = const \times Q(K^c) <\epsilon .$$

If we assume $\int |f(x)|^2 < \infty$ and $Q(x) \leq 1$ then: $$ P(K^c)=\int_{K^c} P(x) dx =\int_{K^c} f(x) Q(x) dx \leq \sqrt{\int_{K^c} |f(x)|^2 dx \times \int_{K^c} Q^2(x) dx} <\epsilon .$$

Please also see: Equivalent ideas of absolute continuity of measures

Let $P_n(x) = f_n(x) Q(x)$ similar to above: Now we have, following bound uniformly: $$D(P_n || Q) = \int f_n(x) \log(f_n(x)) Q(x) dx < v $$ Try proving one of the above assumptions on $f_n(x)$ from above KL divergence inequality uniformly. Let me know if this is useful.

Answer 2

If we additionally assume that $P_n ≪ Q$ for each $n$ the result seems to hold.

Indeed, by conditional Jensen, for any measurable $K$, $$ KL(P_n||Q) = _{P_n}[\ln\frac{dP_n}{dQ}|K^c]P_n[K^c] + _{P_n}[\ln\frac{dP_n}{dQ}|K]P_n[K]\\ = _{P_n}[-\ln\frac{dQ}{dP_n}|K^c]P_n[K^c] + _{P_n}[-\ln\frac{dQ}{dP_n}|K]P_n[K]\\ ≥ -\ln(Q[K^c]/P_n[K^c])P_n[K^c] + -\ln(Q[K]/P_n[K])P_n[K] $$ For $ε > 0$, choose $K$ compact such $Q[K^c] < ε$, $Q[K] ≥ 1-ε$ by the tightness of $Q$. Assume that $P_n$ is not (uniformly) tight. Then, there is $γ > 0$ such that for each compact $K$ there is $n$ with $P_n[K^c] ≥ γ, P_n[K] < 1-γ$. Hence, for each $ε > 0$, there is an $n$ such that $$-\ln(Q[K^c]/P_n[K^c])P_n[K^c] ≥ -\ln(ε/γ)γ,$$ which can be made arbitrarily large by a suitable small $ε > 0$. On the other hand, $-\ln(Q[K]/x)x$ is bounded below in $x∈[0,1]$, so $-\ln(Q[K]/P_n[K])P_n[K]$ is bounded below. But then the bounded-KL condition cannot hold.