Showing KL divergence is $0$ iff two distributions are the equal almost everywhere

4.7k Views Asked by At

It is clear that the KL-divergence $D_\text{KL}(P\parallel Q) = 0$ if $P = Q$ almost everywhere*, for two distributions $P$ and $Q$. However, I don't know how to show the converse, i.e. that if $D_\text{KL}(P\parallel Q) = 0$, then the density functions are equal, i.e. $p(x) = q(x)$ $(P$-)almost everywhere. Since

$$D_\text{KL}(P\parallel Q) = \ln\frac{p(x)}{q(x)} p(x)\,dx $$

and $\ln \frac{p(x)}{q(x)}$ could take on any real value, isn't it possible that the integral could be zero by the cancellation of some negative and positive contributions of the integrand? What would be the correct approach to showing the converse statement?

*(and should this be "with respect to the distribution $P$"?)

1

There are 1 best solutions below

1
On BEST ANSWER

Let $f(t)=-\ln t$. Then $f$ is strictly convex. We have $0=\int f(\frac {q(x)} {p(x)})p(x)\,dx \geq f(\int \frac {q(x)} {p(x)}p(x)\,dx)=f(1)=0$. Since equality holds in Jensen's inequality and $f$ is strictly convex it follows that $\frac {q(x)} {p(x)}$ is a constant, hence $=1$ almost everywhere. Hence $P=Q$.