Coin toss bias estimation problem

Question

Alice and Bob are given a set of five biased coins. They both estimate the probability that each coin will show a head when flipped, and each coin is then flipped once. These are the estimates and values observed:

Coin $1$: Alice's estimate $0.4$, Bob's estimate $0.2$, Observed - Heads

Coin $2$: Alice's estimate $0.7$, Bob's estimate $0.8$, Observed - Heads

Coin $3$: Alice's estimate $0.2$, Bob's estimate $0.3$, Observed - Tails

Coin $4$: Alice's estimate $0.9$, Bob's estimate $0.6$, Observed - Tails

Coin $5$: Alice's estimate $0.4$, Bob's estimate $0.3$, Observed - Heads

Whom would you say is better at estimating the bias of the coins, and why?

I would like guidance on how to approach this question.

Answer 1

One approach is to look at the sum of the squared deviations from the observed outcome ($1$ if heads, $0$ if tails):

$$A: \sqrt{0.6^2+0.3^2+0.2^2+0.9^2+0.6^2}= 1.288\dots$$ $$B: \sqrt{0.8^2+0.2^2+0.3^2+0.6^2+0.7^2}=1.273\dots$$

So, Bob wins, but not significantly!

Another is to count who came the closest the most times, which results in $3:2$ in favor of Alice. This, however, has the same "drawbacks" as for instance the scoring system in Tennis, where one player can actually win more games (which for our example would be equivalent to being closer overall), but still loose the match.

Answer 2

There are several possibilities. Here is one, which uses multiplication rather than your suggestion of addition:

Alice's estimated likelihood of the observed outcome is $0.4 \times 0.7 \times 0.8 \times 0.1 \times 0.4 = 0.00896$

Bob's estimated likelihood of the observed outcome is $0.2 \times 0.8 \times 0.7 \times 0.4 \times 0.3 = 0.01344$

So Bob's estimated likelihood is $1.5$ times Alice's estimated likelihood, suggesting Bob's estimates may have been slightly better. Alice's estimate for Coin $4$ was particularly unfortunate.

But somebody else who thought there was no predictable bias might say $0.5$ each time and have an overall likelihood for the outcome of $0.5^5=0.03125$, more than double Bob's figure, suggesting that neither were particularly good.