Intuition behind the critical difference of ANOVA

Question

I am studying Analysis of Variance. Suppose, we have done ANOVA and found out that null hypothesis is rejected. This means there is a significant difference between at least one pair of the treatments. Now, we would like to know about those pairs. For that my book is calculating something called as Critical Difference between any two pairs:

$CD.=t_{n-k}(\alpha/2).S_{E}^2\sqrt{2/n}$

Here, $n-k$ is the degree of freedom of the error. And $S_{E}^2$ is the sum of square due to error. I am not able to understand the intuition behind the above formula for critical difference.

Answer 1

This is also called Tukeys method (https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm). Think of it like a confidence limit for rejecting the null hypothesis that no pair are different.

Answer 2

This is also called the Fisher least significant difference (LSD). I think by $S_E^2$ they mean $M\!S_E=S_E^2/(N-k)$. Note that the error degrees of freedom is $N-k$, where $N=nk$, assuming $k$ treatments and equal sample sizes of $n.$ Consider a $100(1-\alpha)$ confidence interval for the difference in two means $\mu_1-\mu_2$:

$$(\bar{Y_1}\ - \bar{Y_2})\color{red}\pm t_{N-k,\alpha/2}\sqrt{2M\!S_E/n}.$$

If $$\hspace{16pt}(\bar{Y_1}\ - \bar{Y_2})+t_{N-k,\alpha/2}\sqrt{2 M\!S_E/n}\color{red}{<}0$$ $$\hspace{-60pt}\color{green}{OR}$$

$$\hspace{35pt}(\bar{Y_1}\ - \bar{Y_2})-t_{N-k,\alpha/2}\sqrt{2 M\!S_E/n}\color{red}{>}0\hspace{20pt} $$

then we reject $H_0\!: \mu_1-\mu_2=0.$ Thus, we reject $H_0\!: \mu_1-\mu_2=0$ if and only if $$\hspace{29pt}|\bar{Y_1}\ - \bar{Y_2}|\color{red}>t_{N-k,\alpha/2}\sqrt{2 M\!S_E/n}=CD.$$

Answer 3

That particular formula may not be the best way to inform your intuition. Here is a way to look at significance in terms of a comparison of two variance estimates. (There is a reason this procedure is called analysis of variance.)

ANOVA model. Consider a one-way ANOVA design the $g = 3$ groups and $n = 10$ replications in each group. The model is $$Y_{ij} = \mu + \alpha_i + e_{ij}.$$ for $i = 1,2,3,$ ahd $j = 1, 2, \dots, 10,$ where $e_{ij}$ are a random sample from $\mathsf{Norm}(0, \sigma).$

The null hypothesis is $H_0: \mu_1 = \mu_2 = \mu_3$ and the alternative is that the $\mu_i$ are not all equal.

Two variance estimates. The ANOVA procedure uses two estimates of the variance $\sigma^2$ common to the three groups.

Estimating gariability within groups. The first estimate of $\sigma^2$ uses an extension of the pooled variance in a pooled 2-sample t test. Extended to three groups we have the within-groups variance estimate $$S_w^2 = \frac{(n_1-1)S_1^2 + (n_2-1)S_2^2 + (n_3-1)S_3^2}{n_1 + n_2 + n_3 - 3},$$ where $S_i^2$ is the sample variance of group $i.$ This estimate is an unbiased estimate of $\sigma^2,$ whether or not $H_0$ is true.

Estimating variability among groups if $H_0$ true. If $H_0$ is true, then $\mu_1=\mu_2=\mu_3 =\mu.$ Thus the three means $\bar X_i \sim \mathsf{Norm}(\mu, \sigma/\sqrt{n}.$ If we treat the three $\bar X_i$ as a sample, we can find their sample variance, which is an estimate of $\sigma^2/n$ and multiply it by $n.$ Then $$S_a^2 = n\frac{\sum_{i-1}^g(\bar X_i = \bar{\bar X})^2}{g-1},$$ is another unbiased estimator of $\sigma^2,$ where $\bar{\bar X} =\frac{1}{gn}\sum_{i=1}^g\sum_{j=1}^{n} X_{ij},$ is the grand mean of all $gn$ observations.

Now we have two estimates of $\sigma^2:$ (a) the within groups estimate $S_w^2,$ which is always a good estimator, and (b) the among groups estimate $S_a^2,$ which is good only if $H_0$ is true. (If $H_0$ is false it will tend to be too large.)

Variance ratio as test criterion. Finally, we can see that $F = \frac{S_a^2}{S_w^2}$ will tend to be about $1$ if $H_0$ is true and larger than $1$ if $H_0$ is false. Specifically, $F \sim \mathsf{F}(\nu_1 = g-1, \nu_2 = n(g-1)),$ Snececor's F-distribution with numerator DF $\nu_1$ and denominator DF $\nu_2.$ In the current example, that's $\mathsf{F}(2, 27).$

In an ANOVA table you will often see $S_a^2$ called MS(Group) or MS(Factor) and $S_w^2$ called MS(Error) or MS(Residual).

Intuitive comparison. The intuitive part is that you can look at the numerator $S_a^2$ of $F$ as showing how diverse the $\mu_i$ may be and the denominator $S_w^2$ of $F$ as showing the variability within the samples. If $S_a^2$ is sufficiently larger then $S_w^2,$ then the $F$-statistic will be sufficiently larger than $1$ to reject $H_0.$

Graph of F density with critical value. A graph of the density function of $\mathsf{F}(2, 27)$ with the critical value $c = 4.004$ at the vertical red line. The area under the curve to the right of this line is 5%. In R,

qf(.97, 2,27)
[1] 4.004093

Example. In the following example $S_a^2 = 142.4, S_w^2 = 14.4, F = 9.73 > 4.00,$ so $H_0$ is rejected at the 5% level.

set.seed(2020)
x1 = rnorm(10, 50, 3)
x2 = rnorm(10, 51, 3)
x3 = rnorm(10, 53, 3)
x = c(x1,x2,x3); g = rep(1:3, each=10) 
stripchart(x ~ g, ylim=c(.5, 3.5), pch=20.)

anova(lm(x ~ g))
Analysis of Variance Table

Response: x
          Df Sum Sq Mean Sq F value   Pr(>F)   
g          1 142.38 142.383  9.7345 0.004165
Residuals 28 409.55  14.627