How much of a statistical anomaly would be sufficient to raise suspicion that a probability game is unfair?

Question

It concerns me that nothing is actually impossible in gambling. Even a one in a billion possibility is not impossible. So it makes me wonder how one could analyse data and say that something is so unlikely that there may be an issue with whatever produced the data.

How unlikely does something have to be before it can be said that it shouldn't have happened?

I'm guessing it would need to be a series of unlikely events in quick succession?

I'm far from an expert on statistics but I would like to learn more, have been studying variance, standard deviation and probability with a statistician recently, but many years to go before I can answer this myself.

Answer 1

This is answered by hypothesis testing and is a branch called statistical inference. Hypothesis testing answers exactly this kind of questions.

Based on my sample of results from this game, how likely is it that this sample comes from a fair game?

In simple words: I observed the outcome of the game $n$ times (10 or 20 or whatever) and based on these outcomes, I can determine how likely it is that a fair game produced this sequence of outcomes.

The answer is always given with a margin for error. This margin comes from the fact that you highlight, that in fact anything is possible.

Answer 2

That depends on a lot of factors such as sample size, probability distribution chose and what is called your tolerance/significance level denoted by $\alpha$, a number between 0 and 1 that represents your level of scepticism (0 is absolutely sceptical, 1 is not at all). in my classes, we usually use 0.05. i heard some economists use 0.10 and some engineers use 0.02.

Example: you flip a coin a hundred times and you get all heads. is the coin fair?

If we have an $\alpha = 0.000..(\text{insert a lot})..01$, a hundred is probably not enough to suspect the coin is not fair: theoretically, the coin could be fair and give a hundred flips on the first try (but then again, this depends on what 'probably' means: are using the standard normal probability distribution? a t probability distribution?).

if we have an $\alpha = 0.05$, probably it is not fair

if we have an (economist?) $\alpha = 0.10$, very likely it is not fair

for all $\alpha$'s, if we flip said coin only, say, twice, it is clear, hopefully, that because of the low sample size, there is not enough evidence to say the coin is not fair

Legal analogy: in court, how much evidence is needed to say that a particular defendant is guilty? some people are convicted of murder even though no murder weapon or body is found, I think.

a witch trial might correspond to an $\alpha > 0.2$

a trial that is only for show corresponds to an $\alpha = 0.999$

a murder trial that requires the prosecution to produce a signed confession, a body, a murder weapon, several witnesses and testimonies including expert testimony probably has an $\alpha = 0.001$. A legal expert/statistician will probably give you a less inaccurate $\alpha$. (Got 'signed confession' from Larsen and Marx)

Epistemological analogy: how do we know anything known a posteriori is true eg evolution is true, global warming is due to pollution or the earth revolves around the sun?

Given the evidence accumulated up to this point in time (rather than, say, 2000 years ago)

an $\alpha = 0.000..(\text{insert a lot})..01$ might correspond to suspicion of global scientific conspiracy

an $\alpha = 0.01$ might correspond to 'if mainstream science says so, i'm prepared to act as if it is true'

read more: https://en.wikipedia.org/wiki/Statistical_hypothesis_testing#The_testing_process