How to compare two experiments from the Bayesian perspective?

Question

Imagine that I have two different coin flipping experiments:

• Experiment 1: Uses one coin with M₁ heads and (N₁-M₁) tails.
• Experiment 2: It is composed of two sub-experiments each using a different coin. Experiment 2.1: M₂₁ heads and (N₂₁-M₂₁) tails. Experiment 2.2: M₂₂ heads and (N₂₂-M₂₂) tails.

Note that I can not aggregate M₂₁+M₂₁ because the sub-experiments use different coins and potentially each one could be differently biased.

Obviously as the number of sub-experiments increase the probability of the data conditioned to the hypothesis tends to zero because the domain of the probability increases in dimension. So, it can not be directly compared with experiment 1.

How can I know which experiment better supports/rejects the hypothesis, H₀, that probability of head for all coin/s is 0.5? What is the justification for the proposed solution?

Update: After jorki comments, I realized that it would be helpful to write down my understanding of the first experiment more formally.

In the first case (multiple throws using the same coin), if I'm not mistaken, the probability of the hypothesis coin is fair subject to the data is: $$ p(H_0|M_{11},N_{11}) = \frac{p(M_{11},N_{11}|H_0)}{p(M_{11},N_{11})}p(H_0)=\frac{{N_{11}\choose M_{11}} 0.5^{N_{11}}}{\int_0^1 {N_{11}\choose M_{11}} p^{M_{11}}(1-p)^{N_{11}-M_{11}} dp} p(H_0) = (N_{11}+1)\frac{1}{2^{N_{11}}} p(H_0) $$

Am I doing a mistake? Because I do not understand why the posterior probability of the coins being fair ($H_0$) decreases as the number of throws increases? or otherwise, how the prior should be related to the number of throws?

Answer 1

Obviously as the number of sub-experiments increase the probability of the data conditioned to the hypothesis tends to zero because the domain of the probability increases in dimension.

That's not true. Conditional on the hypothesis that all coins are fair, the probability of the data is simply $2^{-n}$, where $n$ is the total number of coin flips; the number of subexperiments doesn't enter into it. But anyway, you're interested in the posterior probability of the hypothesis, not in the probability of the data conditioned on the hypothesis.

The title says that you want to approach the problem from a Bayesian perspective, but you never mention that again in the text, and in particular you don't mention your prior. If the prior exhibits the phenomenon that you describe, then the posterior probability will also reflect that. But that just means that, given your prior, the hypothesis is less likely to be true when there are more subexperiments.

To illustrate this, let's say there are two different coin makers, Fairie and Leprecoin. Fairie is known for making state-of-the-art ultra-precise fair coins, and their coins can for all intents and purposes be assumed to be exactly fair. Leprecoin sells random bits of metal superficially made to look like coins; their probability to show heads is uniformly randomly distributed on the unit interval.

Let's consider three scenarios:

• $1$) You buy all your coins from Leprecoin.
• $2$) You buy each coin from either company with equal probability.
• $3$) For each of your two experiments, you buy all coins for that experiment from either company with equal probability.

In the first case, your prior is continuous, so the probability for all coins to be fair is zero and remains zero, no matter how often you flip them. You can only define a probability that the coins are all approximately fair; say, that their probability to show heads lies in $[0.49,0.51]$. In this case, already before you start flipping any coins, the probability that the coin in Experiment $1$ is approximately fair is $0.02$ whereas the probability that both coins in Experiment $2$ are approximately fair is only $0.0004$. There's nothing wrong with that; it merely expresses the fact that it's much more likely to buy one approximately fair coin from Leprecoin than two. This will continue to be reflected in the posterior distribution when you conduct the experiments.

In the second case, you do have a non-zero probability for all coins to be fair, but again, before you start flipping, the probability that all coins are fair is already $\frac12$ in Experiment $1$ and $\frac14$ in Experiment $2$, again merely reflecting the fact that if you buy uniformly randomly from the two companies, it's more likely that one coin is bought from Fairie than it is that two coins are bought form Fairie. Again, nothing wrong with that, and again, this will continue to be reflected in the posterior distribution when you conduct the experiments.

In the third case, you do start out with the same probability $\frac12$ that all coins in an experiment are fair. And that will continue be reflected in the posterior distribution when you conduct the experiments.