Lower bounds of the entropy function

125 Views Asked by At

Let $H(x,p)$ be the entropy function. That is $$H(x,p) :=x\log(\frac{x}{p})+(1−x)\log(\frac{1-x}{1-p})\quad(0<x<1,\,0<p<1)$$

A) Show if $x\neq p$ then $H(x,p) \geq 2(x-p)^2$

B) Show if $\frac{1}{2}\leq p < x$ (or if $x<p\leq\frac{1}{2}$) then $H(x,p) \geq \frac{(x-p)^2}{2p(1-p)}$

I can't seem to figure either of these out. In a preceding problem we found the first and second derivative of $H,$ which may factor into the solution, but I cannot see where. Does anyone have any idea ? I think there is something simple I am missing for both here.

Edit: Proof for (A)

First see that for $0<x<1$, $\frac{d^2H}{dx^2} = \frac{1}{x(1-x)}\geq 4$ Now, integrating both sides we see $\frac{dH}{dx} = 4x$, and integrating again we have $H \geq 2x^2 \geq 2(x-p)^2$ since $0<p<1$.

I am fairly sure the same argument applies to part (B). I thought I had it figured out, but while writing I realized my attempt was incorrect. I attempted to do roughly the same exact thing but since $2p(1-p)$ varies from 0 to 1/2, it is larger than $2(x-p)^2$.

1

There are 1 best solutions below

0
On BEST ANSWER

There are some problems with your proof of part (A). First, you probably mean $\frac{dH}{dx} \ge 4x$, not $\frac{dH}{dx} = 4x$. But that conclusion would be valid only if $\frac{dH}{dx}$ is zero for $x=0$, which it isn't: For fixed $p$, the function decreases first and then increases with increasing $x$. The same problem is with the conclusion that $H \geq 2x^2$, and $2x^2 \geq 2(x-p)^2$ does not hold for $x < p/2$.

These problems can be solved. With $h(x) = H(x, p)$ for fixed $p$ you can show that $h(p) = h'(p) = 0$. Then $h''(x) \ge 4$ implies $$ h(x) = h(p) + h'(p) (x-p) + \frac{h''(c)}{2}(x-p)^2\ge 2 (x-p)^2 \, , $$ using Taylor's theorem. You can proceed similarly for part (B): If $\frac{1}{2}\leq p < x$ then $$ h''(t) = \frac{1}{t(1-t)} \ge \frac{1}{p(1-p)} $$ for $p \le t \le x$.

Here is a different approach, using an integral representation, similarly as in Prove Positivity of a Function Involving $\log$..

We have $$ H(x, p) = x\int_p^x \frac{dt}{t} + (1-x)\int_{1-p}^{1-x}\frac{dt}{t} \\ = \int_p^x \left( \frac xt - \frac{1-x}{1-t}\right) \, dt = \int_p^x \frac{x-t}{t(1-t)} \, dt \, . $$

For part (A) we use $t(1-t) \le 1/4$, that gives $$ H(x, p) \ge 4\int_p^x (x-t) \, dt = 2(x-p)^2 \, . $$

(Remark: Yes, that works if $p \le x$ and if $p \ge x$.)

For part (B), if $\frac{1}{2}\leq p < x$ or $x<p\leq\frac{1}{2}$ then $t(1-t) \le p(1-p)$ for all $t$ in the integration interval, and therefore $$ H(x, p) \ge \frac{1}{p(1-p)}\int_p^x (x-t) \, dt = \frac{(x-p)^2}{2p(1-p)} \, . $$