How Do You Call The Probability That An Outcome Is By Chance

Question

I can't find an answer to my question because I don't know what to search for, i.e. the terminology.

To explain the word I'm looking for, let me start with an example. Say we know that the probability for having heads up after a coin flip is 50%. Now say I have a theory that if you flip the coin a certain way, it increases your chance of having the coin land with heads up. I try the theory 100 times (flipping the coin), and have heads up 60% of the time. Now I want to have a metric that meaningfully represents how likely it is that my technique actually increased the the heads-probability, vs. it just being chance.

What do you call this metric?

I'm naturally interested in the maths behind this question, but am rather asking for the terminology only to avoid duplicate questions. Optionally, if this is a "new" question, feel free to also include the maths.

Answer 1

Terminology to search for would be hypothesis testing.

In particular you might find it interesting that we cannot speak of the probability that your technique helped without further assumptions. Depending on your preferred point of view, you might either need a prior -- an assumption on how likely you thought it was that your technique would help before you performed the test -- or you could compute a p-value, which represents how unlikely the result is given that your technique does not help, but this is not a probability.

Answer 2

When people talk about "the probability that $E$ happened by chance" one can get the impression they're talking about $P(E')$, where $E'$ is the event "$E$ happened by chance". Sometimes, worse, it seems like that's what the author meant.

That's mathematically meaningless - $E'$ is not an event. (Meaningless in general, unless we've somehow set up a model where $E'$ is an event.)

What "the probability that E happened by chance" actually means, or should mean, is "the probability of $E$, assuming that $E$ happened by chance". Not the same thing at all.

From your question it sounds like you're asking about the first interpretation. Various "hypothesis tests" give information about the second interpretation.

Answer 3

The original question described a scenario where the following three variables are known:

1. Probability of success on a single trial
2. Number of trials
3. Number of successes (x)

The probability of such events can be represented by a Cumulative Binomial Probability.

Using the original example, the probability of

• given 50% probability of success on a single trial,
• 60 or more heads-up
• in 100 trials

is 2.84%.

It is "cumulative" because in this case it makes sense to take the probability of 60 successes or more, i.e. we perform a summation of binomial probabilities.

Here is a helpful online calculator.