rationalwiki on "Extraordinary claims require extraordinary evidence"

Question

I don't have a strong background in probability/statistics and I'm trying to understand the example at http://rationalwiki.org/wiki/Extraordinary_claims_require_extraordinary_evidence#Probability_theory

Is that the correct framework to explain the principle? Their $P(A)$ there seems arbitrary, and using other values and the Bayes' formula they employ one gets values for $P(A|B)$ greater than one.

I've been trying to work out a more general example in which the experiment consists of tossing $N$ times and the event $B$ would be guessing right $n$ times, but again it seems to me that there must be something limiting $P(A)$ or else one can get values of $P(A|B)$ higher than one.

But again, I don't really know how to make sense of the example in a completely rigorous way or even if that's the correct approach to illustrate the principle.

Thanks!

Answer 1

They did use Baye's formula imprecisely.

Their values assigned to the claim's premise are correct.   $P(A) = 0.000001$ is the assumed measure of the claim, and $P(B\mid A)=0.9$ is the measure of accuracy given the claim.

However, $0.5$ is not the probability measure of the evidence, $P(B),$ as they gave.   It is the measure of the evidence if the claim were not so.   I.E. $P(B\mid \neg A)=0.5$.

So what they should have used was: $$\begin{align} P(A\mid B)\quad & = \frac{P(B\mid A)P(A)}{P(B)} & \text{Baye's Rule} \\[1ex] & = \dfrac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid \neg A)P(\neg A)} & \text{by Law of Total Probability} \\[1ex] & = \dfrac{0.9\times 0.000001}{0.9\times 0.000001 + 0.5\times 0.999999} & \text{from the values in the premise} \\[1ex] & \approx 0.0000017{\small 999986}\ldots \end{align}$$

Which was approximately close enough for their point to hold but, as use noticed, the formulation they used would not hold for less that extraordinary claims.

Answer 2

Bayes' formula can never give $P(A|B) > 1$ when used correctly. The main trouble with using it in applications is that the prior probabilities are subjective.

Suppose, for simplicity, that there are just two alternatives:

($A_1$) The guesser has probability $0.9$ of guessing any given coin-flip correctly (whether this comes from magical powers or cheating is irrelevant).

($A_2$) The guesser has probability $0.5$ of guessing any given coin-flip correctly, as would be the case for random guessing.

Let's say I judge $P(A_1)$ to be $0.000001$ and $P(A_2)$ to be $.999999$.

We then do an experiment, flipping $100$ coins, and find that the guesser was correct $87$ times (this is event $B$). According to the binomial distribution with parameters $100$ and $p$, this has probability ${100 \choose 87} p^{87} (1-p)^{13}$. So $P(B|A_1) \approx 0.074302$ and $P(B|A_2) \approx 5.6 \times 10^{-15}$. According to Bayes,

$$ P(A_1 | B) = \dfrac{P(B|A_1) P(A_1)}{P(B|A_1) P(A_1) + P(B|A_2) P(A_2)} \approx .99999992$$

So this would be pretty convincing.