This is maybe a trivial question, but I don't know the answer right now. Assume you have $n$ coins, each with probability $P_i$ of landing on head. You don't know the probabilities, but there are $m$ agents $A^j$ who claim that the $i$th coin lands on head with probability $P^j_i$. You are only allowed to toss each coin exactly once. Assume you want to somehow decide how trustworthy the agents are. Which scoring function $s:2^n\times[0,1]^n\to\mathbf{R}$ (input tossing result and claimed probabilities, output score) would you use?
Some properties I imagine the scoring function should have:
- $E(s(-,(P'_i)_i))<E(s(-,(P_i)_i))$ for $(P'_i)_i\neq (P_i)_i$
- $P(s(-,(P'_i)_i)<s(-,(P_i)_i))>0.5$ for $(P'_i)_i\neq (P_i)_i$
First thing which comes to mind is computing the probability that the observed tosses occur under the given probability distribution.