Problems to understand the definition of the level of significance $\alpha$

Question

I'm having problems to understand the definition of the level of significance $\alpha$. I thought I knew what $\alpha$ is but I realized I don't.

When I stated to study statistics by myself I read this introductory book and everything was fine, the definition is very clear. He says on page 290:

You’re probably wondering, how small does a p-value have to be for us to achieve statistical significance? If we agree that a p-value of $0.0001$ is clearly statistically significant and a p-value of $0.50$ is not, there must be some point between $0.0001$ and $0.50$ where we cross the threshold between statistical significance and random chance. That point, measuring when something becomes rare enough to be called “unusual,” might vary a lot from person to person. We should agree in advance on a reasonable cutoff point. Statisticians call this cutoff point the significance level of a test and usually denote it with the Greek letter $\alpha$ (alpha). For example, if $\alpha = 0.05$ we say we are doing a $5\%$ test and will call the results statistically significant if the p-value for the sample is smaller than $0.05$. Often, short hand notation such as $P < 0.05$ is used to indicate that the p-value is less than $0.05$, which means the results are significant at a $5\%$ level.

Now, I'm studying about statistical inference, a more advanced subject, and I realized there are some concepts that don't exactly have the same definition as I studied before. The level of significance is an example.

I'm reading this book and on page 352 he introduces the Neyman-Pearson lemma as a method to find the UMP test.

Example:

On the basis of a random sample of size $1$ from the p.d.f. $f(x; \theta)=\theta x^{\theta-1},\ 0 < x < 1\ (\theta > 1)$:

For $\theta_1>\theta_0$ , the cutoff point is calculated by:

... $C=(1−\alpha)^{\frac{1}{\theta_0}}$

For $\theta_1<\theta_0$ , we have:

... $C = \alpha^{\frac{1}{\theta_0}}$

So in this second book, the cutoff point is not necessarily $\alpha$, I'm confused.

MY ATTEMPT TO UNDERSTAND WITH THE HELP OF THE ANSWERS

The alpha is predetermined, but it doesn't mean I can't have a smaller rejection region. Then I end up having a smaller rejection region using NP lemma with the same level of significance alpha. Some introductory books let the cutoff point to be $\alpha$ by standard (why?), that's the reason of my confusion, I can shrink the rejection region keeping the value of $\alpha$. Can someone say if I'm right?

Answer 1

You are misunderstanding what "cutoff" means in the context of the Neyman-Pearson lemma. It is referring to a cutoff of the likelihood ratio in the test, not as a cutoff of what $p$-values are small enough to be significant.

---

A $p$-value (of some results) represents the probability under the null hypothesis of getting results at least as extreme/unusual as your current results. The idea is that seeing results that are unusual under the null hypothesis should be evidence to reject the null hypothesis. To decide how small a $p$-value is small enough to be significant, you must set a significance level, say $\alpha = 0.05$; if your $p$-value is smaller than $\alpha$, this indicates that your data is unusual under the null, so you reject the null.

Comparing $p$-values against $\alpha$ guarantees that your Type I error is $\le \alpha$: if the null is true, the probability that your test is wrong is $\le \alpha$. More generally, this is how you should be thinking about significance levels: as Type I error (probability that a test rejects the null if null is true). This is important for likelihood ratio tests, where there aren't $p$-values being computed anywhere.

---

"Cutoff" in Neyman-Pearson

In this context, you are forming a test based on a statistic called the likelihood ratio $\Lambda(x) = \frac{L_1(\theta \mid x)}{L_0(\theta \mid x)}$. (I am following your textbook's convention of putting the null in the denominator.)

The test is of the form

reject $H_0$ if $\Lambda(x) >C$

do not reject $H_0$ if $\Lambda(x) < C$

for some cutoff $C$. (I'm ignoring the case where the likelihood ratio equals $C$ for simplicity.) The intuition is that if the data $x$ seem to support the alternative hypothesis, the numerator of the likelihood ratio would be large so we would lean toward rejecting; likewise if the data $x$ seem to support the null, the denominator would be large and we would lean toward not rejecting. This is $C$ is what the "cutoff" is in the Neyman-Pearson example. It is not a cutoff for what $p$-values are small enough to be significant.

How do we choose the cutoff? If $C$ is large, then you reject less often (leading to small Type I error); if $C$ is small you reject more often (leading to large Type I error). If you have set a significance level, then you must set $C$ large enough to avoid a large Type I error. The example you are reading is solving for the smallest $C$ that keeps the Type I error below $\alpha$.

Answer 2

First remember that you are dealing with simple hypothesis (both simple).

Applying NP Lemma you get that the critical region is

$$\frac{ \theta_0x^{\theta_0-1} }{ \theta_1x^{\theta_1-1} }\leq k$$

$$x^{\theta_0-\theta_1}\leq k^*$$

Now it is easy to observe that

• if $\theta_0>\theta_1$ the critical region can be viewed as $x\leq c$ and thus, by definition of $\alpha$ you get

$$\int_0^c \theta_0x^{\theta_0-1}dx=\alpha\rightarrow c=\alpha^{1/\theta_0}$$

• if $\theta_0<\theta_1$ with similar reasoning (that I left to you as an exercise) the critical region can be viewed as $x\geq c$ that is $c=(1-\alpha)^{1/\theta_0}$

---

In these formulas $\alpha$ is the given significance level. Usually $5\%$ for a significant test or $1\%$ for a high significant test

---

Definition of $\alpha$

$$\alpha=\mathbb{P}[\mathbf{x}\in C|H_0]$$

Where $C$ is the critical region calculated with NP Lemma

---

I suggest you to read Casella Berger or Mood Graybill Boes that are two reference textbook for these arguments