Is relative entropy with respect to a pmf a continuous function?

Question

Is the relative entropy $D(p || q)$ with a fixed pmf $q$, continuous over $p$, where $p \in \{x \in \mathbb{R}^n: \sum_{i=1}^n x_i = 1 , x_i \geq 0 \}$?

Answer 1

Suppose $q_i > 0$ for all $i\in\{1,\dots, n\}$. Then, assuming (as usual) that $0\log 0=0$ for the definition, we have $$\begin{align} D(p\mid\mid q) &= \sum_{i=1}^n p_i \log_2 \frac{p_i}{q_i} = \sum_{i=1}^n p_i \log_2 \frac{p_i}{q_i} = \sum_{i=1}^n p_i \log_2 \frac{p_i}{q_i} \\ &= \sum_{i=1}^n p_i \log_2 p_i - \sum_{i=1}^n p_i \log_2 q_i \end{align}$$ Both terms are continuous with regard to $p$:

• the second is obvious: it is of the form $\langle p, y\rangle$ for $y\in\mathbb{R}^n$ defined by $y_i = \log_2 q_i$

• the first one because the function $\colon x\in[0,1]\mapsto (x\log_2 x)\mathbb{1}_{(0,1]}(x)$ is continuous.

If there exists $i\in \{1,\dots, n\}$ such that $q_i=0$, it is no longer true. Indeed, then you can have, for any $\varepsilon > 0$, $D(p\mid\mid q) = \infty$ and $D(p^\prime \mid\mid q) < \infty$ while $\lVert p - p^\prime\rVert_1 \leq \varepsilon$.