Question about modeling and interpreting probability in the real world as it pertains to the election and the real world in general

49 Views Asked by At

Like a lot of us have, I have been looking at models that weigh each candidate's odds of winning the upcoming election. Due to the outcome of the last election, there has been a lot of slander on the accuracy of these models since they gave Trump a low chance of winning in 2016. A question I had about evaluating the quality of a model was whether if in 2016, a model that gave Trump a 20% chance of winning was necessarily bad or inaccurate. After all, when broken down into each respective state, he did we win by very slim margins and so the idea of a poll saying he has a low chance of winning seems fair to me. Would a model that said he had an 80% chance of winning necessarily be better since he did end up winning? This seems both intuitive and counterintuitive. For example, if we were to have a model that said a die has a 100% chance of rolling a 4, and a 4 was rolled, I'm not sure this would make the model good, or even accurate.

I guess the question I have is how do we evaluate the accuracy of models after we know the outcome? Is it fair to say these were bad models? What does it mean for the event to occur when if it had a less than 50% chance of happening? Was it actually unlikely to happen, or did we just fail to accurately assess parameters that lead to this outcome?

I'm not sure if this is a stupid question as I'm not really well versed in probability, so sorry if it is, but I was just having trouble reconciling my intuitions on this topic.

1

There are 1 best solutions below

0
On

Probability is about uncertainty.

The standard model for a fair die is that you get each number between $1$ and $6$ with probability $1/6$. If you roll a die once you are certain to get an outcome whose prior probability was just $1/6$, which is well less than $50\%$. That does not make your model "bad". Thinking that a probability of less than $1/2$ is the same thing as impossible makes no sense - it takes all the meaning out of "probability".

Moreover, probability for dice is easy. You can roll a die over and over again and calculate the fraction of time you see each number. That fraction will approach $1/6$ as the number of rolls increases.

But you can't repeat an election over and over again and calculate frequencies. The meaning of probability for one time events is much more subtle. You might say that a model that gave Trump a $20\%$ chance of winning in 2016 is worse than a model that gave him a $40\%$ chance, since he did win. So you might not use that model this time.

The questions you ask about how to evaluate a model that attempts to predict a one time event like an election are good ones. This site is not really the place to answer them. One possible way to start is to use a model to "predict" the outcome of many previous elections, and see how often it was right.

You might consider reading about election models on Andrew Gelman's blog: https://statmodeling.stat.columbia.edu/?s=election+model

And see this from xkcd.