The Taylor expansion of the binary entropy

621 Views Asked by At

Thanks a lot in advance to all the helpers!

I need to bound $H(p)=-p\log_2(p)-(1-p)\log_2(1-p)$ around $q\in(0,1)$ from above with a quadratic function and prove the bound. I thought about doing that (the proof - I manged to get the second order expansion) using the taylor expansion but I can't seem to find it or even a proof of the case $q=\frac{1}{2}$ which appears in Wikipedia (it is $H(p)=1-\frac{1}{2\ln2}\sum_{n=1}^\infty \frac{(1-2p)^{2n}}{n(2n-1)}$).

The taylor expansion, a proof of a quadratic upper bound, or just a reference will help a lot!

edit: Here is what I have so far. I tried to do the expansion myself but I've got results that does not pass the sainty check of substituting $q=\frac{1}{2}$ and comparing to the expansion from Wikipedia. $$H'(x)=\log \frac{1-x}{x}$$ $$H''(x)=\frac{-1}{\ln(2) x(1-x)}$$ For any $i\geq 2$ the $i$'th derivative is $$-\frac{(i-2)!\left((1-x)^{i-1}-(-x)^{i-1}\right)}{\ln(2)x^{i-1}(1-x)^{i-1}}=-\frac{(i-2)!\sum_{k=0}^{i-2}\binom{i-1}{k}(-x)^k}{\ln(2)x^{i-1}(1-x)^{i-1}}$$ We prove the claim by induction. $i=2$ is the base case. To conclude the proof \begin{align} \left(\frac{(1-x)^n-(-x)^n}{x^n(1-x)^n}\right)'&=\frac{n(1-2x)((1-x)^n-(-x)^n) -x(1-x)(-n(1-x)^{n-1}+n(-x)^{n-1})}{x^{n+1}(1-x)^{n+1}}\\ &=n\frac{(1-x)^n(1-2x+x)-(-x)^n(1-2x-1+x)}{x^{n+1}(1-x)^{n+1}}\\ &=n\frac{(1-x)^{n+1}-(-x)^{n+1}}{x^{n+1}(1-x)^{n+1}} \end{align}

Am I missing something? Thanks again to all the helpers!

1

There are 1 best solutions below

0
On BEST ANSWER

Hint

Since you are concerned by what happens around $p=\frac 12$, let $p=\frac 12+q$ and work around $q=0$ the function $$\frac 1{\log(2)}\Big[\frac{1}{2} (2 q-1) \log (1-2 q)-\frac{1}{2} (2 q+1) \log (1+2 q)+\log (2) \Big]$$ and use the Taylor expansion of $\log(1+\epsilon)$.

This is more than simple and will give Wikipedia formula if, when done, you replace $q$ by $(p-\frac 12)$