What causes Type I Errors and Type II Errors to occur?

Question

So in my stats book I am told the following: A type I error is a false positive meaning you reject the null hypothesis when it's true. A type II error is a false negative meaning you reject the alternative hypothesis when it's true. My question is what causes these errors to occur? Is it simply bad sampling methods resulting in your getting biased data that causes you to get a skewed test statistic value and make the wrong conclusions?

What causes these types of errors in hypothesis testing?

Thank you

Answer 1

Both of these possible errors are simply the result of the fact that you are conducting a statistical test. Since chance is involved, your results are not always going to be correct. You can try to design the experiment to minimize the probabilities of these errors, but you'll never eliminate them completely.

Answer 2

Say I was doing a statistical test to determine whether A coin was biased or not. In this case my null hypothesis would be

$$ H_0: P(\mathrm{Heads}) = P(\mathrm{Tails}) = \frac 12$$.

and the alternative hypothesis would be

$$ H_1: P(\mathrm{Heads}) \ne P(\mathrm{Tails})$$

Now say I do 6 coinflips and do not get a single head. Now we calculate the P-value, which is the probability that I get a result, as extreme as the experiment or more extreme, given that the null hypothesis is true. In this case it is the sum of the probability of getting all head or all tails. This is $$\left(\frac12\right)^6+\left(\frac12\right)^6 = \frac{1}{32} = 0.03125$$.

How the standard cutoff for P-values is $P<0.05$. So in this case we would reject the null hypothesis and conclude that the coin is biased. What is important to note however is that the coin may not have been biased, but it could just be random chance. This would be an example of a type 1 error.

In an alternative universe I might have done the the same experiment, but instead have flipped 5 tails and a single head. In this universe I would calculate the P-value to be 0.21875. Since this is bigger than 0.05 I would accept the null hypothesis and conclude that the coin is not biased. However, the coin could still be biased and flip heads once. In this case I made a Type 2 error.

Answer 3

There can be a few causes, even if the test is properly run.

We can get any type of error by bad luck. Suppose we want to test if a coin is fair by flipping it 1 thousand times. If we got only 5 heads, it's still technically possible that it is a fair coin and we just got an outcome that statistically should occur less than once every googol times we run the test. Similarly, it's possible that we got exactly 500 heads with a coin that is weighted so that the chance of a heads is only $0.02$, an outcome that should occur only about 31 out of every googol times.

We also often get false negatives because the test doesn't have good enough resolution. If we take a null hypothesis that a medication has no effect, but its actual effect is very small, we would need a very large sample in order to have any chance of dependably giving the true positive result; it's possible that the required sample size is too large to be theoretically possible. Giving a numerical example, suppose a rare disease affects one million people worldwide, with a mortality rate of 5%. We want to test if a new medication will help them, but that new medication actually only reduces the mortality rate by $0.0001$%. Over the entire infected population, it will save a total of one person, but a difference of one person dying isn't enough to be testable