How can I check if three or more independent normal samples are statistically identical?

Question

I conducted several experiments (for the moment 3, but there will be more in the near future), where in each experiment I made several measurements. I know (from previous experience) that the result roughly normally. For each experiment I have the mean value and the standard deviation of the results. I am trying to test the hypothesis that all three results are identical, with p% accuracy. When I had only 2 results I used the t-test and found that there was no significant difference between them, but now that I have 3 I am a bit lost.

Could you please help with this?

Thanks!

Answer 1

It seems the procedure you are looking for is a one-factor (or 'one-way') analysis of variance (ANOVA) with three levels of the factor.

The model is $$Y_{ij} = \mu_i + e_{ij},$$ for $i = 1, 2, \dots g\,$ (levels), $j = 1, 2, \dots, n_i$ (replications at each level), and $e_{ij}$ are independently distributed as $\mathsf{Norm}(0, \sigma).$ In the 'balanced case', the numbers $n_i = n$ of replications are all the same.

In case $g = 3,$ one tests $H_0: \mu_1 = \mu_2 = \mu_3$ against the alternative hypothesis that the $\mu_i$ are not all equal.

Notice that one assumes the data to be normally distributed and the the variance in each level (or group) to be the same. Formulas for finding 'degrees of freedom', 'sums of squares', 'mean squares', and the F-statistic for testing the null hypothesis are given in many elementary applied statistics textbooks.

The rationale is to find two estimates of $\sigma^2,$ called MS(Factor) and MS(Error). [MS stamds for 'mean square'.] MS(Error) is an accurate estimate of $\sigma^2$ whether or not $H_0$ is true. By contrast, MS(Factor) is accurate if $H_0$ is true, but tends to be too large if $H_0$ is false. Thus, $F = $MS(Factor)/MS(Error) tends to be large (roughly larger than about 1) when $H_0$ is not true. If the assumptions of normality and equal variances are true, then $F$ has an F-distribution with the appropriate numerator and denominator degrees of freedom. This provides the desired test of hypothesis.

Below is an example of data for a one-factor ANOVA with computations in Minitab statistical software (slightly edited for relevance). I used $g = 3, n = 10.$ And the data were sampled (and rounded to integers) from normal distributions with $\mu_1 = \mu_2 = 30, \mu_3 = 40, \sigma = 2,$ so that $H_0$ is false.

Method

Null hypothesis         All means are equal
Alternative hypothesis  At least one mean is different
Significance level      α = 0.05

Equal variances were assumed for the analysis.


Factor Information

Factor  Levels  Values
Factor       3  A, B, C


Analysis of Variance

Source  DF      SS       MS  F-Value  P-Value
Factor   2   663.2  331.600    59.06    0.000
Error   27   151.6    5.615
Total   29   814.8

Means

Level of
Factor   N    Mean  StDev       95% CI
 A      10  29.200  1.814  (27.663, 30.737)
 B      10  30.600  2.547  (29.063, 32.137)
 C      10  39.800  2.658  (38.263, 41.337)

Pooled StDev = 2.36956

The very small P-value printed as 0.000 (meaning $< 0.0005)$ indicates that not all three group mean are equal. [This is hardly a surprise because of the way I simulated the data.]

The confidence intervals (each with confidence 95% separately) suggest that Group C differs from the other two. A formal 'multiple comparison' procedure would allow testing this with a specified 'overall' error rate.

Data:

A
   32   27   27   31   30   30   27   30   30   28

B
   27   28   29   34   32   29   31   29   33   34

C
   41   38   42   43   35   38   42   40   37   42