Proof of nonnegativity of KL divergence using Jensen's inequality

Question

I'm a bit confused by the proof:

$KL(p||q) = -\int p(x) \log\left\{\frac{q(x)}{p(x)}\right\}dx \ge -\log \int p(x) \frac{q(x)}{p(x)}dx = -\log \int q(x)dx = 0$

where the first inequality is the application of the Jensen's inequality:

$f(E(x)) \le E(f(x))$

The argument was $\log(x)$ is a concave function - yes, I know that; but isn't the function here $\log\left\{\frac{q(x)}{p(x)}\right\}$, not $\log(x)$? And $p(x)$ is the density of $x$, not that of $\frac{q(x)}{p(x)}$? How can we apply Jensen's inequality here?

Answer 1

That follows from a rather trivial generalization of Jensen inequality:

Let $f,g:\mathbb{R} \to \mathbb{R}$ with $f(\cdot)$ convex. Then $E[f(g(X))] \ge f (E[g(X)])$

The proof is simple: apply the Jensen inequality to the random variable $Y=g(X)$. Notice that no convexity condition (actually, no condition at all) is required for the function $g$. But also notice that it's only the (convex) function $f$ the one that "goes outside the expectation" in the inequality.

In your case, take $f(x) = \log(x)$ (concave) and $g(x)=q(x)/p(x)$ (further: don't let the fact that in $g(x)=q(x)/p(x)$ $q$ and $p$ are densities confuse you; that does not matter at all).

Answer 2

A little reminder - there seems to be a minor fault in your proof. Since $log(x)$ is concave, it follows that $\mathbb{E}(f(x)) \leq f(\mathbb{E}(x))$, not the other way around.

You can denote $f(x) = -\log(x)$ here as a whole to make it convex though, so that $\mathbb{E}(f(x)) \geq f(\mathbb{E}(x))$ and your argument still holds.