You conduct a study and acquire a p value = 0.15. The Null hypothesis (Ho) is Mu = Sample. (Ha) is Mu != sample. Alpha = 0. 05. Does this mean that:
A. There is a 15% chance the null hypothesis is true
B. There is a 15% probability that the sample is due to random chance
C. if more samples are taken there is a 15% chance that they will be as extreme as this sample
D. None of these are true
E. A - C are true.
The answer is C. The significance level is supposed to be the cut off for the likelihood of the null hypothesis being incorrect. If this is true, why aren't A and B correct?
A. In the classical frequentist framework (don't worry if you don't understand what that means; it's basically the current universal default for intro-level stats), the null hypothesis is either true or it's not. You don't know which one it is, but that doesn't make it appropriate to talk about it probabilistically. Either the average height of all humans is 5.5 feet, or it isn't, but it's not sensible to place a probability on that just because you don't know the statement's truth value.
B. EDIT: I actually don't like the original answer I gave here, so here's a better one instead. It doesn't make sense to talk about "the probability that a sample was due to random chance." The sample was presumably a random sample, so it was due to random chance, full stop. The correct answer should be something more like, "the probability that results as extreme (or more so) as what was observed would arise due to random chance if the null hypothesis is true." Notice a key detail here; the probability is on the results on the experiment, not on the "sample" itself. As stated, answer B parses quite strangely as an English sentence.
Even if answer B said, "There is a 15% probability that the results of the experiment are due to random chance," it still wouldn't be good; that probably necessarily depends on whether or not the null hypothesis is true. A correct answer should explicitly state that the null hypothesis is assumed to be true when calculating the P-value. Note that in reality, it still might be false!
As Brian Borchers pointed out in the comments, the assumption that $H-0$ is true is notably missing from C as well, so it too should be regarded as false.