A problem asked me to explain why a $99\%$ confidence interval of the difference of two sample means did not offer substantial evidence of a difference in the two populations. The interval was something like $(1,11)$ and clearly didn't include $0$.
I was under the impression that if $0$ is an unlikely difference value, then there is evidence that the two population means are different.
What am I not understanding? Did I possibly overlook some essential info?
Your concern is appropriate. A CI for the difference that does not include zero provides statistical evidence that the two means are different. This is true when some assumptions are verified, i.e. the two populations have homogeneous variances, the populations are normally distributed, and each value is sampled independently from each other value.
If no mention of these assumptions is specified in the problem, a possible justification for not having 'substantial evidence" of different means despite the lack of zero in the CI might depend on violations of these assumptions.
It should also be clarified whether the authors of the problem gave to the word "substantial" a particular meaning (e.g. in terms of difference magnitude, and so on).