Information theory applied to a dice (intuition about information theory)

Question

Let's say I have a die that has the values 1 to 6 written on it, and I don't know the probability to get each value, I only know that when I throw the die many times I get an average value of 3.5, just like a fair die.

According to information theory, I can guess the most unbiased probability to get a certain value by maximizing the uncertainty under the constraints, where the uncertainty is:

$$H=−K\sum_{i} p_i\ln(p_i)$$ and the constraints are:

$$\sum_{i}p_i=1$$ and

$$\sum_{i}p_i⋅v_i=3.5$$

where $v_i$ are the values of the dice between 1 to 6. If I maximize $H$ under the constrains I get:

$$p_i∝\exp(−v_iμ/K)$$

Where μ is the Lagrange multiplier corresponding to the second condition.

My question is: Is this really the most unbiased probability we can find under the constraints?

The possibility of equal probabilities ($p_i=1/6$) fulfills the constraints, and according to Occam's razor principle, it should be more likely than exponential probability. What am I missing?

Answer 1

In general, for a given mean $m$, we have the restrictions $\sum_{i=1}^6 p_i=1$ and $\sum_{i=1}^6 p_i i = m$. Applying Lagrange multipliers we get for the critical point: $ -1 - \log p_i + \lambda i +\beta=0$

Hence the extrema is given by a (truncated) exponentional family $$p_i = a \exp({-b i}) $$ where the constants are given by the restrictions.

Now, in the particular case where $m=(1+6)/2$, you'd get $b=0$ and $a=1/6$, which amounts to an uniform distribution. (You can deduce this, with no need of doing the calculation by Stelios' comment: the uniform distribution gives the maximum entropy without the mean restriction, and your particular restriction is fullfilled by that distribution.)

Hence, there is no contradicion here, because the uniform distribution indeed belongs to the exponential family.

BTW: I used the expression "maximum entropy distribution" because "most unbiased probability" is rather confusing ("unbiased" has another meaning in statistic) and "more likely (according to Occam's razor principle)" can also be misleading ("likely - lilelihood" also has a definite meaning - perhaps one should better say "more preferable")

Answer 2

The comments are spot on, but the problem is in general a little tricky unless $n$ is very large. Cover and Thomas' Information Theory text discuss this problem in their maximum entropy chapter, apparently it was first discussed by Boltzmann who started with the uniform distribution and tried to argue this would lead to maximum entropy.

Assume you throw $n$ dice on a table with the total number of "spots" showing equal to $n\alpha,$ how to find the most probable macrostate?

If $n_i$ is the number of times face $i$ is seen for $1\leq i\leq 6,$ then there are $$\binom{n}{n_1,n_2,n_3,n_4,n_5,n_6}$$ ways this can happen, and for uniform distribution the probability of seeing these $n_i$ is $$6^{-n}\binom{n}{n_1,n_2,n_3,n_4,n_5,n_6}.$$

We can maximize the probability by maximizing the multinomial coefficient under the constraint $\sum_{i=1}^6 i n_i=n\alpha.$ Using a crude Stirling approximation $n!\approx (n/e)^n,$ gives $$\binom{n}{n_1,n_2,n_3,n_4,n_5,n_6}\approx \frac{(\frac{n}{e})^n}{\Pi_{i=1}^6 (\frac{n}{e})^{n_i}}=\Pi_{i=1}^6 \left(\frac{n}{n_i}\right)^{n_i}$$ and the right hand side can be rewritten to give $$\binom{n}{n_1,n_2,n_3,n_4,n_5,n_6}\approx\exp\left[H\left(\frac{n_1}{n},\frac{n_2}{n},\frac{n_1}{n},\frac{n_2}{n},\frac{n_1}{n},\frac{n_2}{n}\right)\right] $$ where the exponent needs to be maximized using Lagrange multipliers, and results in the maximizing distribution $$ p_i^{\ast}=\frac{e^{\mu i}}{\sum_{i=1}^6 e^{\mu i}}. $$ For $n=6,$ and $E(X)=3.5,$ solving this respecting the mean constraint is equivalent to solving $$ e^{\mu}+2 e^{2\mu}+3 e^{3 \mu} + 4 e^{4 \mu}+ 5 e^{5 \mu}+ 6 e^{6 \mu} = 3.5(e^{\mu}+ e^{2\mu}+ e^{3 \mu} + e^{4 \mu}+ e^{5 \mu}+ e^{6 \mu}) $$ or $$ -2.5 e^{\mu}-1.5 e^{2\mu}-0.5 e^{3 \mu} + 0.5 e^{4 \mu}+ 1.5 e^{5 \mu}+ 2.5 e^{6 \mu} = 0, $$ and letting $e^{\mu}=Z$ and solving the resulting polynomial will give a root at $Z=1,$ i.e.,$\mu=0,$ as in the comments.

For a general mean, the rate of convergence in the crude stirling will determine the correspondence between the maximum probability distribution and maximum entropy distribution.