p-value, intuition about type-I error=$\alpha$

Question

In hypothesis testing consider this situation:

Define $\alpha=P\{\text{type I error}\}=P\{\text{ Rejecting } { H_0}\text{ when } H_0\text{ is true}\}$.

Also define $p-$value like this

What is the probability of observing a data which is similar to the one on hand, or more extreme, if $H_0$ happens to be true. This probability will be called the $p-$value.

How do I get some inutition about this: A test with a small $p-$value indicates that the null hypothesis is less plausible than the alternative hypothesis and in this case $H_0$ is rejected.

Answer 1

Two-sided test. Suppose you have the following ten observations from a normal population, and do a two -sided test $H_0: \mu = 100$ against the two sided alternative $H_1: \mu \ne 100.$

 102, 114, 114,  80,  83,  79, 117,  79, 115,  79

In R statistical software, the test looks like this:

t.test(x, mu=100)

        One Sample t-test

data:  x
t = -0.68402, df = 9, p-value = 0.5112
alternative hypothesis: true mean is not equal to 100
sample estimates:
mean of x 
     96.2 

Under the null hypothesis, the test statistic $T$ has Student's t distribution with 9 degrees of freedom. The observed P-value would be the probability of getting a value from that distribution that is less than $-0.684$ or is greater than $0.684,$ which is just as far away from 100, but in a positive direction. [In R, a CDF of Student's t distribution is denoted pt.]

 pt(-0.684, 9) + (1 - pt(0.684, 9)) 
 ## 0.5112009

In the figure below, you want the sum of the two probabilities (areas) that lie outside the vertical broken lines.

One-sided test. Now suppose you test $H_0: \mu = 100$ against the one-sided alternative $H_0: \mu < 100.$ (Same data as above.)

        One Sample t-test

data:  x
t = -0.68402, df = 9, p-value = 0.2556
alternative hypothesis: true mean is less than 100
sample estimates:
mean of x 
     96.2 

Now the P-value is only the probability in the left tail of the Student's t distribution, to the left of $-.684.$ (Because of the symmetry of the t distribution, this turns out to be half as large as the P-value for the two-sided test.)

pt(-0.684, 9)
## 0.2556005

In the figure, the P-value is only the probability to the left of the vertical broken line.

Note about 'wrong sided' tests: If you had been unfortunate enough to choose the 'wrong' side in framing the alternative, testing $H_0: \mu = 100$ against the one-sided alternative $H_0: \mu > 100,$ then the P-value is the area to the right of the vertical broken line in the second figure. Key lines from the output of the t test in R would be as follows:

t = -0.68402, df = 9, p-value = 0.7444
alternative hypothesis: true mean is greater than 100

And using the CDF of Student's t distribution:

1 - pt(-.684, 9)
## 0.7443995