Area Under The Curve of CDF and Alpha Value - Chi-Sq Test Interpretation

Question

Can you help me understand the meaning of the area under the curve of a PDF in the interpretation of a Chi-Square test?

This is what I think I know:

If my test statistic (i.e. the chi-square value) increases for a given Degree of Freedom, so does my cumulative distribution function (CDF) value of the Chi-Square CDF(Chi). This value represents the area under the curve of the Chi-Square probability density function (PDF).

CDF(Chi) is also the significance level or "critical region" called $\alpha$ (often used with 0.05).

Is it correct, that if for a given test value the CDF(Chi) > 0.05 my H0 can be rejected when only interested in a one-tailed test - i.e. are two distributions different or not? This approach seems to work if I do a one-tailed test.

This is however currently counterintuitive to me, because as far as I understand, p = 1 - CDF(Chi) and H0 is rejected if p < 0.05. Therefore I would believe that the above statement is not true and I would need to reject H0 if CDF > 0.95. That would also make more sense to me, as it means that H0 is only rejected if there is more than 95% "evidence" that it is not random or less than a 5% chance of wrongfully rejecting H0.

Can someone help clear things up - I assume my last thought makes more sense and in my example, I am getting results that seem to conform to what I want to see but are actually not statically significant.

Thanks!

Answer 1

Consider the following fictitious counts for four categories, where the first category is half as likely as the other three. Thus, we would suspect that, with enough observations, a chi-squared test of the null hypothesis--that categories are equally likely--will be rejected.

set.seed(205)
x = sample(1:4, 100, rep=T, p=c(1,2,2,2))
table(x)
x
 1  2  3  4 
14 34 25 27 

Notice that the first category does indeed have a lower count out of 100 observations than do the other three categories.

With 100 observations under the null hypothesis, the expected count for each category is $25.$ You can check for yourself that the chi-squared statistic is $8.24.$ (One of the observed counts is 'too small' and three are 'too large', but differences are squared, so all four terms in the statistic are positive: $\sum_{i=1}^4 \frac{(X_i-25)^2}{25} = 8.24.)$

Because the chi-squared statistic is approximately distributed as $\mathsf{Chisq}(\nu = 4-1=3)$ when $H_0$ is true, the critical value of the test is $c = 7.815,$ which cuts probability 5% from the upper tail of $\mathsf{Chisq}(\nu = 3),$

Because the chi=squared statistic is $8.24 > 7.815,$ the agreement between observed and expected counts is worse than would be expected by chance, and the null hypothesis is rejected at the 5% level of significance. (Remember that the chi-squared statistic really measures badness of fit: the larger the statistic, the worse the fit of the data to the null hypothesis.)

 c = qchisq(.95, 3);  c
 [1] 7.814728

The P-value of this chi-squared test is $0.041,$ the probability in the right tail of $\mathsf{Chisq}(\nu=3)$ beyond the observed value $8.24.$ We reject the null hypothesis at the 5% level if the P-value is smaller than $0.05.$

1 - pchisq(8.24, 3)
[1] 0.04130349

In R, the formal chi-squared test looks like this. (When ho hypothetical probabilities are specified, the default null hypothesis is that all categories are equally likely.)

chisq.test(table(x))

        Chi-squared test 
        for given probabilities

data:  table(x)
X-squared = 8.24, df = 3, p-value = 0.0413

The figure below shows the density function of $\mathsf{Chisq}(\nu 3).$ The vertical blue line is at the observed value of the chi-square statistic; the area to the right of it under the curve is the P-value. The vertical dotted red line is at the critical value $c;$ the area to the right of it under the density curve is 5%.