How to interpret when A>C is statistically significant in t-test, but neither A>B nor B>C is statistically significant?

Question

I hope this probably basic level of statistics also passes as mathematics, though the specific example I use is biological. I am confused about the meaning of statistical significance/insignificance when there are multiple parties involved.

Let’s say there are three cancer drugs, A, B, and C. We have three groups of people in the trial, each treated with one of the above drugs. At the end of the trial, we measure the sizes of tumors; the more effective the drug is, the smaller the number is.

We obtained these results:

The average of A is lower than the average of C, and it is statistically significant in t-test, p<0.05. So we think A is a better drug than C. The average of A is also lower than the average of B, but it is NOT statistically significant in t-test: p>0.05. So we think A is not a better drug than C. The average of B is lower than the average of C, but it is NOT statistically significant in t-test: p>0.05. So it seems B is not a better drug than C.

So how can A be not better than B, and B is not better than C, but A is better than C?

Can I conclude that A is better than C indeed?

Answer 1

If an overall one-factor ANOVA (with three levels of the factor) rejects the null hypothesis that not all three population means are equal, there is no guarantee that post hoc analysis will completely resolve which means are equal and which are not.

You have found an instance where the sample means are in the order $\bar X_A < \bar X_B < \bar X_C.$ And you have evidence at a chosen level of significance to conclude that the difference between $\bar X_A$ and $\bar X_C$ is significant. So you suppose that $\mu_A < \mu_C.$

However, you are simply unable to resolve whether the smaller difference between $\bar X_B$ and $\bar X_A$ is significant. Similarly, for the smaller difference between $\bar X_C$ and $\bar X_B.$

So you can only speculate whether $\mu_A < \mu_B < \mu_C$ or $\mu_A \le \mu_B < \mu_C$ or $\mu_A < \mu_B \le \mu_C.$ Worse than that, it could even be that $\mu_B < \mu_A$ or $\mu_B > \mu_C.$ The quantities $\bar X_A,\,\bar X_B,\,$ and $\bar X_C$ are only estimates of $\mu_A,\, \mu_B$ and $\mu_C.$

Be happy for the differences among population means that can be "resolved" (at some level of significance) by ad hoc tests, and try not to be annoyed by non-significant differences.

Example: Here is an example of a one-way ANOVA with five levels of the factor. The main F-test finds significant difference(s) among the levels, but Tukey HSD ad hoc test can resolve only that the two extremes are significantly different (marked by my #).

Simulated data:

set.seed(2020)
sg=3; n = 15
x1 = rnorm(n, 100, sg)
x2 = rnorm(n, 101, sg)
x3 = rnorm(n, 102, sg)
x4 = rnorm(n, 103, sg)
x5 = rnorm(n, 104, sg)
round(rowMeans(rbind(x1,x2,x3,x4,x5)),3)
     x1      x2      x3      x4      x5 
 99.944 101.997 102.105 103.001 105.114 

ANOVA

x = c(x1,x2,x3,x4,x5)
g = as.factor(rep(1:5,each=n))
aov.out = aov(lm(x ~ g))
summary(aov.out)

             Df Sum Sq Mean Sq F value  Pr(>F)   
g            4  210.1   52.51     4.7 0.00204 **
Residuals   70  782.1   11.17                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Tukey HSD Comparisons.

TukeyHSD(aov.out)
Tukey multiple comparisons of means
  95% family-wise confidence level

Fit: aov(formula = lm(x ~ g))

$g
         diff        lwr      upr     p adj
2-1 2.0529529 -1.3646567 5.470562 0.4515460
3-1 2.1617703 -1.2558393 5.579380 0.3983888
4-1 3.0575552 -0.3600544 6.475165 0.1011180
5-1 5.1700035  1.7523939 8.587613 0.0006348  # Levels 1 & 5 signif diff
3-2 0.1088175 -3.3087921 3.526427 0.9999853
4-2 1.0046023 -2.4130073 4.422212 0.9226291
5-2 3.1170507 -0.3005589 6.534660 0.0905099
4-3 0.8957848 -2.5218248 4.313394 0.9478189
5-3 3.0082332 -0.4093764 6.425843 0.1106571
5-4 2.1124484 -1.3051612 5.530058 0.4221567