How to prove that $\mathrm{Prob}(\sum_{i=1}^nX_i \leq np) \leq 2^{-nD(p||q)}$?

66 Views Asked by At

Let $X_1, \dots, X_n$ be binary valued independent and identically distributed random variables with common distribution $Q=(Q(0),Q(1))=(1-q,q)$, and let $p \leq q$.

I would like to prove that $$\mathrm{Prob}(\sum_{i=1}^nX_i \leq np) \leq 2^{-nD(p||q)},$$ where $$D(p||q)=p \log \frac{p}{q}+(1-p) \log \frac{1-p}{1-q}.$$

I suspect that it has something to do with Sanov’s theorem.

Here is what I have tried:

First, I have tried to prove $$\mathrm{Prob}(X_i=1 | \sum_{i=1}^n X_i \leq np)\leq p,$$ and in order to prove the above I tried to determine $$\mathrm{Prob}(X_i=1 | \sum_{i=1}^n X_i=k), \, 0 \leq k \leq np.$$

Unfortunately, I got stuck.