I have a sample question for an exam and would like some help understanding if I'm approaching it correctly. The question:
We toss three fair (unbiased) coins.
For each of them, heads and tails have the identical probability 1/2.
The resulting random value is a vector hx1, x2, x3i where xi ∈ {H, T}.
1. Compute entropy of this random variable.
2. How does the entropy change if one of the three coins is unfair (biased) and it always lands with heads up?
With a fair coin there are 8 possible outcomes/vectors: HHH, HHT, HTH, HTT, TTT, TTH, THT, THH Since the coins are fair, each is equally likely, so we have $H(X) = log_28 = 3$
Then if one coin is biased to heads, the outcomes are HHH, HHT, HTH, HTT, TTH, THT, THH
and the calculation of the entropy changes to $H(X) = log_27 ≈ 2.81$, so the entropy decreases slightly.
Is this understanding correct?
This is not correct, because outcomes (assuming the position of bad coin is chosen uniformly) do not have equal probability.
For example, outcome $HTT$ has probability $1/12$: it requires the first coin to be bad ($1/3$) and rolling tale twice ($1/4$). However, outcome $HHT$ has probability $2/12$ (it allows two positions for bad coin).
Overall, we have $3$ outcomes with probability $1/12$, $3$ outcomes with probability $2/12$ and one outcome ($HHH$) with probability $3/12$, for total entropy
$$-\frac{3}{12} \log_2 \frac{1}{12} -\frac{6}{12} \log_2 \frac{2}{12} - \frac{3}{12} \log_2 \frac{3}{12} \approx 2.69$$