I am reading Elements of Information Theory (2nd Ed.) by Cover. I have something that I couldn't figure out.
On page 248, theorem 8.3.1
If the density f(x) of the random variable X is Riemann integrable, then $H(X^Δ)+log(Δ)→h(f)=h(X)$, as $Δ→0$.
Thus, the entropy of an n-bit quantization of a continuous random variable X is approximately h(X) + n.
The continuous random variable X is divided into bins of length $\Delta$.
$X^Δ=x_i\quad if\quad i\Delta\le X<(i+1)\Delta$. Where $\Delta f(x_i)=\int_{i\Delta}^{(i+1)\Delta}f(x)dx$.
The book gives a proof of the theorem, but what I didn't understand is how the last conclusion is reached. Clearly, if the range of X is [0, 1), the length of the bin $\Delta$ is related to n. $-log_2(\Delta)=n$ if the range is [0, 1). But this condition is not required in the theorem. Of course, if the range is finite, when $\Delta$ approaches 0, $-log_2(\Delta) \rightarrow n$. Also, the equation only holds when $\Delta$ is close to 0, which is also not stated in the conclusion. ("Thus, the entropy of an n-bit quantization of a continuous random variable X is approximately h(X) + n.")
Did I miss something, or is there some other proofs of that statement?
Thanks a lot.
In the book, by a "$n-$bit quantization" they mean how many binary (fractional) digits of precision we have.
When we speak in decimal base, we say we have "5 decimal digits" of precision (or that 5 decimal fractional digits are meaningful) when we quantize the real line in intervals of length $1/10000=1/10^5$. (Granted, the word precision is also used with different meanings; often it means the amount of significant digits). Similarly, to have 5 bits of precision (or a $5-$bit quantization) we must quantize the real range in intervals of length $\Delta=1/2^5$. Hence, indeed, $-\log_2 \Delta = n$.
See also the second item in example 8.3.1.
I'm not sure if this answers your doubt.