Gibbs' inequality; continuous case

Question

I'm reading alternative proof of Gibbs' inequality written in wikipedia, which states that

Suppose that $P=\{p_1,...,p_n\}$ be probability distribution.
Then for any other probability distribution $Q=\{q_1,...,q_n\}$, $-\sum p_i\log{p_i} \leq -\sum p_i\log{q_i}$

I have two question.
First, the proof uses Jensen's inequality to claim that $\sum p_i \log \frac{q_i}{p_i} \leq \log \sum p_i \frac{q_i}{p_i}$.
But why does it hold? I think Jensen's inequality just says that $\sum \log{(p_i \frac{q_i}{p_i})} \leq \log \sum p_i \frac{q_i}{p_i}$

Second, can we apply Gibbs' inequality in the continuous case? i.e., does $-\int f(x)\log{f(x)} dx \leq -\int f(x)\log{g(x)} dx$ still holds for probability density function $f, g$?

I think since we can apply Jensen's inequality in the continuous case, we can still argue that the continuous case holds. Nevertheless, I cannot find any mention of Gibbs' inequality in the continuous case; they only deal with the discrete case. Is there any problem to deal with the continuous case? Or can I use Gibbs' inequality in the continuous case as I write?

Thanks.

Answer 1

Partial answer for the first part using "elementary stuff" that OP has asked for in comments:

symbol	explantion
$p_i$	weights
$\log$	concave function
$q_i/p_i$	points in the domain where the concave function is defined

In the quoted inequality $$\sum p_i \log \frac{q_i}{p_i} \leq \log \sum p_i \frac{q_i}{p_i},$$

• LHS: weighted average of $\log(q_i/p_i)$'s.
• RHS: $\log$ of weighted average of $q_i/p_i$'s.

Note that $\log$ is concave, so Jensen's inequality would give the desired result.

The second part is addressed in the linked blog article, and I'm not going retype his/her arguments.

Answer 2

This is a very long date post, but I would like to answer because I am currently facing the same problem. The continuous version of Gibb's inequality is a consequence of Jensen's inequality.

Let $f(x)$ be the true density of $X$ and $g(x)$ an arbitrary density function. Applying Jensen's inequality to $\dfrac{g(x)}{f(x)}$ and since the function $\log$ is concave, we have $E\left(\log\left(\dfrac{g(x)}{f(x)}\right)\right) \leq \log\left(E\left(\dfrac{g(x)}{f(x)}\right)\right) = \log\left(\int \dfrac{g(x)}{f(x)} f(x) dx \right) = \log\Big(\underbrace{\int g(x) dx}_{=1}\Big) $ = 0. Besides, $E\left(\log\left(\dfrac{g(x)}{f(x)}\right)\right) = E\left(\int\log\left(\dfrac{g(x)}{f(x)}\right) f(x) dx\right) = \int f(x)\log\left(g(x)\right) dx - \int f(x)\log\left(f(x)\right) dx$. As a result,

$$\int f(x)\log\left(f(x)\right) \geq \int g(x)\log\left(f(x)\right),$$

which is the continuous version of the Gibbs' Inequality