What does "more extreme" mean when your test statistic does not "support" either hypothesis?

Question

Consider the following example.

My null hypothesis is $p = 0.5.$ My alternative hypothesis is $p > 0.5.$ But when I take a random sample, I the sample proportion is 0.48, with a sample size of 1000.

According to my calculator, the p-value of this is $~0.9.$ In other words, if you draw out the normal distribution of the samples of size $1000$ of $p,$ it shades from the left of the mean, $p,$ all the way to the right.

However, some of those values are not more extreme; for example, 0.49 has a smaller standard deviation from the mean than 0.48. So why is that area still shaded?

Answer 1

First, without the technical details, you are testing whether P(Heads) may exceed $1/2.$ But your sample shows less than half Heads. So you certainly have no evidence that the coin is biased in favor of Heads. In view of your data, you should not be surprised if you get a large P-value when testing the null hypothesis that the coin is fair against a right-sided alternative.

Now for a more formal look at your question. Let $\theta = P(Heads).$ Here is Minitab printout for an approximate normal test of $H_0: \theta = 0.5$ against $H_a: \theta > .5,$ based on observing $X = 480$ Heads in $n = 1000$ tosses of a fair coin.

 Test and CI for One Proportion 

 Test of p = 0.5 vs p > 0.5

 Sample    X     N  Sample p  95% Lower Bound  Z-Value  P-Value
 1       480  1000  0.480000         0.454013    -1.26    0.897

At the 5% level of significance, you would reject $H_0$ in favor of $H_a,$ if the test statistic $Z \ge 1.645.$ [One says that 1.645 is the critical value of the test at the 5% level.] Clearly this is not so, and you cannot reject $H_0.$ A P-value smaller the 0.05 would result if $Z \ge 1.645.$

The P-value is the probability of a more extreme value of the test statistic in the direction of the alternative than the value observed.

Your test statistic is $Z = \frac{\hat \theta - .5}{.5(.5)/1000} = -1.26.$ As you say, the P-value is the area under a standard normal curve to the right of -1.26, and that area is nearly $0.90$ as the Minitab printout shows. In the figure below, this is the area to the right of the vertical dashed line. Such a large P-value is often a signal that something is wrong.

Whenever you get a P-value above about 90%, you should check to see if everything makes sense.

• Are the data a random sample from the assumed distribution?
• Are the null and alternative hypotheses properly formulated?
• If the alternative is one-sided, does the direction match the data? (Here the data don't match the direction of $H_a.$)
• Are you using the software (or tables) correctly?

For a more productive approach, here are some questions to investigate:

(a) If $H_a: \theta \ne 1/2,$ can you reject $H_0: \theta = 1/2$ at level $\alpha = 5\%?$ [Ans: No.] What are the critical values? $[Ans: \pm 1.96.]$ And, what is the P-value?

(b) If $H_a: \theta \le 1/2,$ can you reject $H_0: \theta = 1/2$ at level $\alpha = 5\%?$ What is the critical value? $[Ans: -1.645.]$ And, what is the P-value?