Bayes Theorem Example in Nate Silver's The Signal and the Noise

Question

In his book The Signal and the Noise, Nate Silver presents this example application of Bayes's Theorem on pp. 247-248:

Consider a somber example: the September 11 attacks. Most of us would have assigned almost no probability to terrorists crashing planes into buildings in Manhattan when we woke up that morning. But we recognized that a terror attack was an obvious possibility once the first plane hit the World Trade Center. And we had no doubt we were being attacked once the second tower was hit. Bayes's theorem can replicate this result.

You can view the complete example in Amazon.com's previw, and I've made the two pages available here.

Silver assumes the prior probability of a terrorist plane attack to be 1 in 20,000. After the first plane crash, using Bayes's Theorem he updates that to 38%. And after the second plane crash, he comes up with a 99.99% probability. However, I think he may be mistaken. I'll provide the details below.

To be precise, let us define the following three events:

• $PC$ = Plane Crash: At least one plane crashes into a Manhattan skyscraper on a given day.
• $TPA$ = Terrorist Plane Attack: At least one plane is intentionally crashed into a Manhattan skyscraper on a given day.
• $APC$ = Accidental Plane Crash: At least one plane is accidentally crashed into a Manhattan skyscraper on a given day.

We assume all plane crashes into buildings are either terrorist plane attacks or accidental (i.e. $PC = TPA \cup APC$). Using historical data, Silver estimates the prior probability of an accidental plane crash to be 1 in 12,500. In summary: $$P(TPA) = \frac{1}{20000},$$$$P(APC) = \frac{1}{12500}.$$

Furthermore, Silver assumes $P(APC) = P(PC|\overline{TPA})$ (which is true if $APC$ and $TPA$ are independent events).

Applying Bayes's Theorem, he comes up with $$\begin{align}P(TPA|PC) &= \frac{P(PC|TPA) \times P(TPA)}{P(PC|TPA) \times P(TPA) + P(PC|\overline{TPA})(1-P(TPA))} \\ &= \frac{1 \times \frac{1}{20000}}{1 \times \frac{1}{20000} + \frac{1}{12500} \times (1 - \frac{1}{20000})} = 0.385\end{align}$$

Silver continues:

The idea behind Bayes's theorem, however, is not that we update our probability estimates just once. Instead, we do so continuously as new evidence presents itself to us. Thus our posterior probability of a terror attack after the first plane hit, 38 percent, becomes our prior probability before the second one did. And if you go through the calculation again, to reflect the second plane hitting the World Trade Center, the probability that we were under attack becomes a near-certainty -- 99.99 percent.

That is (this is Silver's calculation): $$P(TPA|PC) = \frac{1 \times 0.385}{1 \times 0.385 + \frac{1}{12500}(1-0.385)} = 99.99 \%$$

"Cool!" I thought, until I thought a bit more. The problem is that you can apply the same logic to calculate the conditional probability of an accidental crash, too. I'll spare you the math, but I come up with $P(APC|PC) = 0.615$ after the first crash, and $P(APC|PC) = 99.997\%$ after the second.

So we can be almost certain the second plane crash is a terrorist attack, and we can be even more certain that it's accidental?

I think the problem is that when Silver applies Bayes's Theorem after the second crash, he uses the updated probability of a terrorist plane attack as his prior, but fails to update the prior probability of an accidental plane crash (which should become 0.615). After the second crash, then, the correct formula is $$P(TPA|PC) = \frac{1 \times 0.385}{1 \times 0.385 + 0.615(1-0.385)} = 0.504$$

Similarly, the probability that we're observing an accidental crash given that there have been two crashes is $$P(APC|PC) = \frac{1 \times 0.615}{1 \times 0.615 + 0.385(1-0.615)} = 0.806$$

Question 1: Am I correct that Nate Silver is doing it wrong?

Question 2: Am I doing it right?

Answer 1

So we can be almost certain the second plane crash is a terrorist attack, and we can be even more certain that it's accidental?

Correct, there is no contradiction here.

If we know that the first crash was a terrorist attack, then the second crash would be more likely another terrorist attack.

The same reasoning with accidental crashes.

Question 1: Am I correct that Nate Silver is doing it wrong? Question 2: Am I doing it right?

No. There is no need to update the rate of accidental crashes. IMHO, Nate implies that accidental crashes don't include terrorist ones. Otherwise, he couldn't multiply probabilities in the denominator.

Answer 2

though the chance of two accidental plane crashes can multiplied to give you a very small number (1/12500 x 1/12500) since they are independent, one cannot assume the same for a terrorist attack. Once we think that the first plane crash is a TPA, it would not make sense to assume that the second crash, if it is also a TPA, to be independent and not highly correlated (perhaps 90% chance that the 2nd plane crash is TPA given the first is TPA) to the first. so if you use 1/20000 x 0.9 to get the probability that both plane crashes are TPA, you will not end up with the problem you mentioned that both scenarios have become more likely.

Answer 3

P(TPA_1/PC)=0.38 P(TPA_2/TPA_1)=0.9 (if 1st plane crash is TPA, 2nd plane crash is almost surely TPA since the two events are highly correlated) P(APC_1/PC)=0.62 P(APC_2/APC_1)=1/12500 (on a bright sunny day, accidental plane crashes have to be independent of each other) hence probability that it is a terrorist attack when the 2nd plane crashed= P(TPC_2/TPC_1)P(TPC_1/PC)/ (P(TPC_2/TPC_1) P(TPC1/PC) + P(APC_2/APC_1) P(APC_1/PC) ) = (0.9 X 0.38)/(0.9X0.38+ 0.62X 1/125000)=99%

Answer 4

I think there is a misunderstanding of Bayes' theorem here. When you update the probability, you do so because you know for sure what has happened the first time; you are not guessing. In the example, Nate Silver assumes, after the first crash, that it was terrorist caused. With that assumption, you go and update and get a new probability. So, the meaning of the 99.9% is that "IF" the first crash was terrorist caused, then there is almost certainty that if there is a 2nd one, it will also be terrorist caused. Now, if the first was accidental, and later there is a second one, you can also be almost certain that the 2nd was another accident.