Comparing P-value and Type 1 Error to reject the null hypothesis

Question

I know, by definition, what p-value and type 1 error mean. However, I have hard time relating those two concepts in rejecting a null hypothesis.

Below are my understanding about P-value and Type 1 Error

1)A p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct. (from wiki)

2)A type 1 error incorrectly rejects a true null hypothesis. That is, reject the Null when in fact the Null is true.

My question is - Why do we reject the null hypothesis when p-value < type 1 error? What are some intuitions behind it? What am I missing..? After learning 1 year of statistics, I still have no idea how this works..

Thanks.

Answer 1

Discrete example (one-tailed test): $T \sim \mathsf{Pois}(\lambda).$ Test $H_0: \lambda = 10$ vs. $H_a: \lambda > 10.$

Because $P(T \ge 16\,|\,\lambda=10) = 1 - P(T \le 15) = 0.0487$ a test at significance level $\alpha = 0.0487 = 4.87\%$ rejects $H_0: \lambda = 10$ vs. $H_a: \lambda > 10$ when $T \ge c = 15,$ where $c$ is called the critical value of the test. Computation in R, where ppois is a Poisson CDF.

1 - ppois(15,10)
[1] 0.0487404

Thus, if you observe $T = 20 > c,$ then you will Reject $H_0$ at level $\alpha.$

However, if you observe $T = 20,$ then the P-value is the probability $P(X \ge 20 \,|\,\lambda = 10)$ $=1-P(X \le 19\,|\,\lambda = 10) = 0.0035,$ and you can say your reject at level $\alpha$ because the P-value is less than $\alpha.$

1 - ppois(19, 10)
[1] 0.003454342

x = 0:30;  pdf = dpois(x, 10)
plot(x, pdf, type="h", lwd=2, ylab="PDF", main="POIS(10)")
 abline(v=0, col="green2"); abline(h=0, col="green2")
 abline(v = 14.5, col="red", lwd=2, lty="dotted")
 abline(v = 19.5, lwd=2)

The sum of the heights of the bars in the plot to to right of the vertical dotted line at the critical value is the significance level $\alpha.$ The P-value is the sum of the heights of the bars to the right of the observed value $T = 20$ (solid line).

Continuous example (two-tailed test): Suppose we have a sample x of $n = 20$ observations from a normal population with unknown mean and variance. We wish to test $H_0: \mu = 100$ against $H_a: \mu \ne 100,$ at the 5% level.

The data are summarized in R as follows:

summary(x); length(x);  sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   81.0    97.5   109.5   107.2   115.2   139.0 
[1] 20           # sample size
[1] 13.94453     # sample SD

In R, a t.test in this situation is as shown below. The sample mean $\bar X = 107.15$ is greater than the hypothetical mean $\mu = 100.$ The question is whether this is sufficiently different to warrant rejecting $H_0.$ According to the t test, the P-value is $0.033 < 0.05 - 5\%,$ so we reject $H_0$ at the 5% level.

t.test(x, mu = 100)

        One Sample t-test

data:  x
t = 2.2931, df = 19, p-value = 0.03342
alternative hypothesis: true mean is not equal to 100
95 percent confidence interval:
 100.6238 113.6762
sample estimates:
mean of x 
   107.15 

Computer output does not always give the critical value. In this particular example the test statistic is distributed as Student's t distribution with $\nu = 19$ degrees of freedom. The critical values for this two-tailed test would be $\pm 2.093,$ where $2.093$ cuts area $0.025$ from the upper tail of $\mathsf{T}(\nu=10),$ as computed in R below (or obtainable from printed tables of t distributions). Knowing the 5% critical values, we can see that $H_0$ is rejected at the 5% level because observed $T = 2.2931$ does not lie between $\pm 2.093.$

qt(.975, 19)
[1] 2.093024

Computer out usually shows a P-value (which can be used to decide whether to reject at any desired level of significance). In the example above, the observed value of the t statistic is $T = 2.2931$ (which you can check by hand from the summary statistics above). The P-value is the probability of an outcome as or more extreme (in either direction from 0). It is computed $0.0341$ as shown in R below.

Knowing that the P-value is smaller than 5%, we can say that $H_0$ is rejected at the 5% level.

[Typically, one can roughly approximate the P-value from printed tables, but exact P-values are ordinarily computed by software. The very small difference below from the P-value in the output is due to rounding error; the output rounds the observed value of $T$ to four places.]

2*pt(-2.2831, 19)
[1] 0.03411225

In the figure below, the significance level $\alpha = 0.05$ is the sum of the two tail areas outside the vertical dotted lines. The P-value is the area to the right of the solid black vertical line, plus the the area to the left of the dashed line on the left (just as far from 0 on the other side).

curve(dt(x, 10), -4,4, lwd=2, ylab="PDF", xlab="t", 
      main="Density of T(19)")
 abline(h=0, col="green2");  abline(v=0, col="greenw")
 abline(v=c(-2.093,2.093), lwd=2, col="red", lty="dotted")
 abline(v = 2.2931, lwd=2)
 abline(v = -2.2932, lwd=2, col="grey", lty="dashed")

Note: The data used to make the sample for the second example is shown below. Even though data were sampled from $\mathsf{Norm}(101, 15),$ the sample mean turned out to be $\bar X = 107.2,$ which is not surprising given the small sample size.

set.seed(522)
 x = round(rnorm(20, 101, 15))

Answer 2

If you just compare the definitions as you provided them, it should be clear why there is a relationship between the $p$-value and the Type I error $\alpha$. If the $p$-value is less than $\alpha$, what this means is that the chance of having observed such an extreme sample when the null hypothesis is true, is less than the Type I error, which is the chance of incorrectly rejecting the null hypothesis. In other words, when $p < \alpha$, the conclusion is to reject $H_0$ based on the evidence. That does not mean $H_0$ is actually false. There is always some probability that $H_0$ is true but the data that was gathered, by chance, was extreme enough to cause you to incorrectly conclude otherwise. Your criteria for rejecting is based on how tolerant you are of this possibility, and is expressed as $\alpha$.

For instance, if I give you a fair coin--we know it is perfectly fair--and you flip it $n = 10$ times, it is still entirely possible, although unlikely, that you obtain $10$ heads in a row. This probability is $\frac{1}{1024}$ even for a fair coin. It would be higher if the coin were biased towards heads, but the point is that even when the null hypothesis (i.e. the coin is fair) is true, random chance allows for "extreme" outcomes that would suggest otherwise if we did not already have knowledge of the coin's properties. Thus, the Type I error $\alpha$ is a way for us to quantify how extreme a result we are willing to accept in order to conclude the coin is not fair, and the $p$-value is the calculation of the probability of having obtained such an extreme result (or more extreme) by pure random chance when the coin is in fact fair.