So, I'm reading about this property in the MacKay book. But I don't fully get it. Can someone explain it to me? There's this example:
A source produces a character $x$ from the alphabet $A = \{0, 1, \ldots, 9, $a$, $b$, . . . , $z$\}$; with probability $\frac 1 3$, $x$ is a numeral $\{0, \ldots, 9\}$; with probability $\frac 1 3$, $x$ is a vowel $\{$a, e, i, o, u$\}$; and with probability $\frac 1 3$ it’s one of the $21$ consonants. All numerals are equiprobable, and the same goes for vowels and consonants. Estimate the entropy of $X$.
Solution. $\log 3 + \frac 1 3(\log 10 + \log 5 + \log 21) = \log 3 + \frac 1 3\log 1050 = \log 30$ bits.
But I don't understand how they calculate it...
Thank you.
A set of $n$ equiprobable events with total probability $p$ contributes
$$ \sum_ip_i\log p_i=\sum_i\frac pn\log\frac pn=p\log\frac pn=p\log p-p\log n $$
to the negative entropy. In your case, all three sets have total probability $p=\frac13$.