p-value and t-distribution

Question

Given a data set $(x_1,\ldots, x_n)$, ( for $n$ - large ) which is realization of a random sample $(X_1, \ldots , X_n)$. Assume the null hypothesis $H_0:\mu = \mu_0$ and alternative hypothesis $H_1 : \mu > \mu_1$ . Then which is an appropriate method of testing whether or not to reject $H_0$?

One way would be to use $t$-test method and find $\pm t_{n-1,\alpha/2}$ ( if $\alpha$ - the significance level is given ) and check where our current observation $T_0 = (X_{\text{average}} - \mu_0)/(S_n/\sqrt n)$ lies. But I also thought about computing the probability that $T$ is at least as extreme as our current our observation ( i.e just compute the $p$-value $P(T > T_0)$. My question is when to use p-value and when t-distribution as a motivation for a decision whether to reject $H_0$.

Answer 1

Once you select the type-I error level ($\alpha$) the two approaches are strictly equivalent. This is explained in most introductory stats books.

For a one-sided test like yours: $$ t>t_{n-1,\alpha} \text{ iff } p<\alpha. $$

Answer 2

Both methods use the $\mathsf{T}(\nu = n-1),$ Student's t distribution with $n - 1$ degrees of freedom.

Background. Suppose we have a sample $X_1, X_2, \dots, X_n$ from a normal distribution with unknown population mean $\mu$ and standard deviation $\sigma.$ We want to use the data to test $H_0: \mu = \mu_0$ against $H_a: \mu > \mu_0.$

We will reject $H_0$ when the statistic $T = \frac{\bar X - \mu_0}{S/\sqrt{n}}$ is sufficiently large. Here $\bar X$ is the sample mean (estimating $\mu)$ and $S$ is the sample standard deviation (estimating $\sigma).$

If the null hypothesis is true, then $T \sim \mathsf{T}(n-1).$

Method 1: Fixed significance level. uses a fixed significance level $\alpha,$ often 0.05. Then one can use printed tables of the t distribution or statistical software to find the critical value $c$ that cuts probability from the upper tail of $\mathsf{T}(n-1).$

For example, if $\alpha = 0.05$ and $n = 15$ one can find that $c = 1.761.$ You should look at a printed t table (row $\nu = 14, perhaps, depending on the style of the table, column headed 0.05 or 0.95) to find this value. The computation from R statistical software is shown below:

qt(.95, 14)
[1] 1.76131

Continuing the example, suppose that $\mu_0 = 20, \bar X = 22.50,$ and $S = 3.75.$ Thus $$T = \frac{\bar X - \mu_0}{S/\sqrt{n}} = \frac{22.50 - 20}{3.75/\sqrt{15}} = 2.582 > c = 1.7613,$$ so that we reject $H_0.$

Method 2: P-value. The P-value is the probability of the occurrence of a value of $T$ that is as extreme, or more extreme (in the direction of the alternative), than the observed value $T = 2.582.$ This probability is calculated using the distribution $\mathsf{T}(n-1).$ However, P-values require statistical software. Only is a few special cases can you get exact P-values from printed tables.) The computation of the P-value in this example using R statistical software is shown below:

1 - pt(2.582, 14)
[1] 0.01086239

Here is how the P-value is used in deciding whether to reject $H_0:$ If we choose to test at level $\alpha = 0.05,$ then we reject $H_0$ if the P-value is smaller than 0.05, which it is.

Also, if we want to test at level $\alpha = 0.02,$ then we rejject because the P-value $= 0.011 < 0.02.$ By contrast, we cannot (quite) reject at level $\alpha = 0.01$ because the P-value (just slightly) exceeds 0.01.$

Graphical summary. Here is a plot of the density curve of the distribution $\mathsf{T}(n-1).$ The critical value $c = 1.7613$ is shown as a vertical dotted line. The area under the density curve to the right of this line is the significance level $0.05 = 5\%.$ The observed value 2.582 of the $T$-statistic is shown as a solid vertical line. The (tiny) area under the curve to the right of this line is the P-value.

This test in Minitab statistical software. The printout below (slightly edited for relevance) shows how Minitab statistical software reports results of this test. Notice that here, as in most statistical software, the output provides a P-value, but not a critical value for any particular significance level.

One-Sample T 

Test of μ = 20 vs > 20


 N     Mean   StDev   SE Mean      T       P
15   22.500   3.750     0.968   2.58   0.011