About Jensen’s inequality and Fano’s inequality

67 Views Asked by At

Let $X, X^′$ be independent with $X ∼ p(x)$, and $X^′ ∼ q(x)$, $x,x^′ \in \mathcal{X}$ . Then:

1.$Pr[X = X^′] ≥ 2^{−H(p)−D(p||q)}$,

2.$Pr[X = X^′] ≥ 2^{−H(q)−D(q||p)}$.

I don't kown how to prove 1 and 2. I think it's likely to use Jensen's inequality. And the Fano inequality has a similar corollary: if $X$ and $ X^ {'} $ i.i.d. with entropy $H(X)$, then $Pr[X = X^′] ≥ 2^{−H(x)} $. I'm not sure whether they're related.

1

There are 1 best solutions below

1
On BEST ANSWER

Hint: Apply Jensen to the (concave) $\log$ function:

$$ \log \sum_x p(x)q(x) = \log E(q(X)) \ge E[ \log q(X)] = \sum_x p(x) \log q(x)$$

Can you go on from here?