Information Inequality theorem

240 Views Asked by At

I was reading the information inequality theorem and got stuck trying to understand something. The highlighted portion says (2.85) becomes an equality if and only if $\frac{q(x)}{p(x)}$ is a constant $c$. Since the sum of $p(x)$ and $q(x)$ must each equal to 1, does that mean that the constant $c = 1$?

Because if $c$ takes any other value then either $p(x)$ or $q(x)$ will not sum to 1. I am also unable to prove that $(2.84) \implies (2.85)$ if $c \neq 1$.

enter image description here

1

There are 1 best solutions below

4
On BEST ANSWER

Note that $p(x)$ and $q(x)$ are probability mass functions. Therefore, $\sum_{x \in \mathcal{X}} p(x)=\sum_{x \in \mathcal{X}} q(x)=1$. $D(p||q)=0$ if the equalities hold in both (2.85) and (2.87). According to (2.85), $q(x)=c p(x)$ for all $x \in A$, whereas (2.87) implies $\sum_{x\in{\mathcal{X}}}q(x)=\sum_{x\in A}q(x)=1$. Therefore, $$1= \sum_{x \in A} q(x) = c\sum_{x \in A} p(x)=c \iff c=1.$$