What's the intuition for Bayes Coherency

63 Views Asked by At

Given an event $E$ and a list of observations $M1,M2,...,Mn$. we can update the probabilities after every observation eg.

$P'(A) = P(A|M1)$ - probability after the first event is seen

$P'(A) = P(A|M2)$ - probability after the first two events are seen.

...

$P'(A) = P(A|Mn)$ - probability after nth event is seen.

This is equivalent with $P(E|M1{\cap}M2{\cap}...{\cap}Mn)$.

It somehow makes sense but I cannot find a general intuition for it.

If you take a simple example where you're trying to find the probability of somebody having cancer given that they smoke and drink, you can either compute the probability of having cancer given they smoke, followed by the probability of having cancer given they drink OR altogether the probability of having cancer given they smoke and drink.

1

There are 1 best solutions below

0
On

The Bayesian update rule is:

$$P(A \mid M_1) = {P(M_1 \mid A) \over P(M_1)} P(A)$$

To generalize it to two "observations" the correct rule is:

$$P(A \mid M_1, M_2) = {P(M_2 \mid A, M_1) \over P(M_2 \mid M_1)} P(A \mid M_1)= {P(M_2 \mid A, M_1) \over P(M_2 \mid M_1)} {P(M_1 \mid A) \over P(M_1)} P(A)$$

which you can easily verify by expanding all the conditional probabilities. Note that this is not the same as the following wrong rule:

$$P(A \mid M_1, M_2) \overset{wrong!}= {P(M_2 \mid A) \over P(M_2)} P(A \mid M_1)= {P(M_2 \mid A) \over P(M_2)} {P(M_1 \mid A) \over P(M_1)} P(A)$$

(There are conditions under which the wrong rule gives the same result as the correct rule, but generally they don't agree.)

So when you said:

compute the probability of having cancer given they smoke, followed by the probability of having cancer given they drink

it really depends on what you mean by "followed by" - and if you meant the second rule, then that's wrong.

Here's a simple example. Roll a fair $6$-sided die and let $A$ be the event that you rolled a $2$. Let $M_1$ be the event ("observation") that the roll is even and $M_2$ be the event that roll is $\le 3$.

  • $P(A) = \frac16$

  • $P(A \mid M_1) = {1 \over 1/2} P(A) = \frac13$

  • Correct update rule: $P(A \mid M_1, M_2) = {1 \over 1/3} P(A \mid M_1) = 1$

  • Wrong update rule: $P(A \mid M_1, M_2) \overset{wrong!}= {1 \over 1/2} P(A \mid M_1) = \frac23$