Why $p$ factor in $p \log p$ in the entropy formula?

Question

It is explained that we have log for additivity of information in the entropy formula. But, why is the $p$ factor? It is redundant, since we already have it in the $\log p$!

Answer 1

From Shannon's original paper ["A Mathematical Theory of Communication", The Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948].

If there is such a measure, say $H(p_1,p_2,...,p_n)$ , it is reasonable to require of it the following properties:

1. $H$ should be continuous in $p_i$

2. If all the $p_i$ are equal, $p_i = 1/n$ , then $H$ should be a monotonic increasing function of $n$. With equally likely events there is more choice, or uncertainty, when there are more possible events.

3. If a choice be broken down into two successive choices, the original $H$ should be the weighted sum of the individual values of $H$.

Basically requirement (3) entails the first $p$. Moverover, it can be shown that the form $H(\{p_i\}_{i=1,...n})=-\sum_i p_i \log p_i$ is the only one that satisfies the above 3 propositions.

Hope it helps.