My stat book contains a section on hypothesis testing. At the beginning of this section, level of significance is defined as the maximum allowable probability of rejecting the null hypothesis when it is true; level of significance is denoted by alpha.
In the next section the test is described: compute the value of p, “the probability of obtaining a sample statistic with a value as extreme or more extreme than the one determined from the sample data.” If p is less than alpha, we reject the null hypothesis.
I feel I’m missing something here. In the first place we defined alpha as the maximum allowable probability of rejecting a true null; and then we used it as a sort of threshold of plausibility. Why are these two ideas interchangeable? They seem different to me.
Take a one-sample t test for example. You have a random sample $X_1, X_2, \dots, X_n$ of size $n=20$ from $\mathsf{Norm}(\mu, \sigma),$ where $\mu$ and $\sigma$ are both unknown. Suppose we test $H_0: \mu = \mu_0$ against $H_a: \mu > \mu_0$ at the 5% level of significance, so that $\alpha = .05.$
The sample mean $\bar X$ estimates the population mean $\mu,$ and the sample standard deviation $S$ estimates the population SD $\sigma.$ The observed value of the t statistic is
$$t_{obs} = \frac{\bar X - \mu_0}{S/\sqrt{n}}.$$
'Under $H_0$' (statistical language for 'assuming the null hypothesis to be true'), $t_{obs}$ is a particular observation from $\mathsf{T}(\nu=19)$ Student's t distribution with $\nu = n-1 = 20-1 = 19$ degrees of freedom.
We reject $H_0$ at level $\alpha = .05$ if $t_{obs} \ge t^* = 1.729.$ This 'critical value' $t^*$ can be found by software or in a table of Student's t distribution (on row 19). It is chosen to cut 5% of the area from the upper tail of the density curve of $\mathsf{T}(\nu = 19).$ Then we have $$P(T \ge t^*\;|\; H_0 \text{ True}) = \alpha = .05,$$ where $T \sim \mathsf{T}(19).$
Now, for the P-value, which I will denote by $p$. It is the probability under $H_0$ that $T$ would take a value greater than $t_{obs}.$ That is, $$P(T \ge t_{obs}\;|\; H_0 \text{ True}) = p.$$
P-values are an innovation of the computer age. Ordinarily, is not possible to find the exact P-value of a t test from printed tables of $\mathsf{T}.$
The sketch below shows the PDF of $\mathsf{T}(19)$. Suppose your value $t_{obs} = 1.87,$ based on the $\bar X$ and $S$ from your sample of $X_i$'s. It is represented by the vertical blue line. The P-value is the area to the right of this vertical blue line.
The solid vertical dotted red line is at $t^* = 1.729,$ the critical value for a test at level $\alpha = .05.$ Because $t_{obs} > t^*$ we reject $H_0$ at the 5% level. From R statistical software I can find the P-value corresponding to $t_{obs} = 1.87$ to be about $p = 0.385,$ but you will not find this value in an ordinary t table.
Because the P-value is smaller than .05, I can tell from the P-value that I should reject $H_0$ at the 5% level.
The vertical dotted orange line is at the critical value (2.539) for a test at the 1% level, which you can find in a printed t table. Because the P-value is larger than .01, I cannot reject $H_0$ at the 1% level.
If my t table shows only critical values for tests at levels 5% and 1%, I can see from the t table that the P-value must be bracketed by .05 and .01, but need software to tell that the P-value is .0385.