How do we find first order estimate of the given entropy of the following image

Question

What is the solution for this answer? Should we use log base 2 in the formula?

Answer 1

You can see the pixels of images as the realization of a discrete random variable on sample space $\Omega = \{0, 1, \ldots, 255\}$ and probability mass function $p \colon \Omega \in [0,1]$ defined as: $$ p(x) = \begin{cases} \frac{5}{16} & \text{if } x = 200 \text{ or } x = 210,\\ \frac{3}{16} & \text{if } x = 215 \text{ or } x = 217,\\ 0 & \text{otherwise}. \end{cases} $$ By applying the Shannon's definition of entropy we obtain: $$ \begin{align} H(X) &= -\sum_{i=0}^{255} p(i)\cdot \log_b(p(i))\\ &= -\frac{5}{16}\log_b\left(\frac{5}{16}\right)-\frac{5}{16}\log_b\left(\frac{5}{16}\right)-\frac{3}{16}\log_b\left(\frac{3}{16}\right)-\frac{3}{16}\log_b\left(\frac{3}{16}\right)\\ &= -\frac{1}{8}\left(5\log_b\left(\frac{5}{16}\right)-3\log_b\left(\frac{3}{16}\right)\right)\\ &= -\frac{1}{8}\left(\log_b\left(\frac{5^5}{16^5}\frac{3^3}{16^3}\right)\right)\\ &= \frac{1}{8}\left(\log_b\left(\frac{16^8}{5^5 3^3}\right)\right)\\ &= \log_b\left(\frac{16}{\sqrt[8]{5^5 3^3}}\right)\\ \end{align} $$ The choice of the value for the basis $b$ of logarithm depends on the units in which you want to express the result:

• base $2$: $ H(X) \approx 1.954$ bits,
• base $e$: $ H(X) \approx 1.355$ nats,
• base $3$: $ H(X) \approx 1.233$ trits,
• base $10$: $ H(X) \approx 0.588$ dits.