Testing whether the two population means are the same using sampling distribution of difference between two means

Question

The problem is

Given the above data, can we conclude that the two population means are equal?

And my question is, how can I solve this question using the sampling distribution of the difference between two means?

I found the variance for the difference of two means:

variance = $\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}$ = (variance of sample 1) + (variance of sample 2) = 125.5 + 104.5 = 230

And since the test is whether or not their means are the same, the assumption would be that the population means are the same. So the mean for the sampling distribution would be $\mu_1-\mu_2=0$.

But from here I got confused because the data provides the 'averages' (which is from the samples I think), and I'm not sure what to do with the sample means. Also, I was planning to find P$((\text{sample mean 1} - \text{sample mean 2})\gt \text{variance}))$ and if this probability is large, conclude that the population means are not the same. But then there's the problem that their variances aren't homogeneous. How can I proceed from here? (by only using the methods of the sampling distribution and not by hypothesis testing!)

Answer 1

A first remark: The formula for the variance of the difference in the two means you give is correct (if we assume independent sampling), but your calculations are wrong. $\sigma_1^2$ is the variance of the distribution from which sample 1 was drawn and $\sigma_2^2$ is the variance of the distribution from which sample 2 was drawn. Those two numbers are unknown. Hence, the variance of the difference in the sample means $\hat{\mu}_1-\hat{\mu}_2$ $$\frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2}\tag{1}$$ is unkown.

The sample variance $\hat{\sigma}_i^2$ is an estimator for $\sigma_i^2$, $i=1,2$. So, we could use the following unbiased and consistent estimator for (1): $$\frac{\hat{\sigma}_1^2}{n_1}+\frac{\hat{\sigma}^2}{n_2} = \frac{125.5^2}{5}+\frac{104.5^2}{6}\approx4970\neq 230.$$ The standard error is hence 70.5.

---

Now on to your question: If you assume that your data is drawn from two independent normal distributions or your sample sizes are large enough to rely on the CLT, then the Welch-test given in heropup's answer is the way to go. Because your sample sizes are very small, I offer another approach using permutation resampling (see resampling), which does not require any distributional assumptions.

The idea is to take your 11 data points and looping through the ${11 \choose 6}=462$ possibilities to put them in two groups of size 5 and 6. For each of them, we calculate the difference in means. This gives us an estimate of the sampling distribution of the difference in means, which you then can you use to calculate the probability you are interested in.

An implementation in R (not very efficient for larger sample sizes) is given by the following code:

library(gtools)  # needed for combinations
x <- c(8260, 8130,8350,8070,8340)
y <- c(7950,7890,7900,8140,7920,7840)
perm_test <- function(x,y)
{
  n <- length(x)
  m <- length(y)
  ncomb <- choose(n+m,n)  # number of combinations
  dta <- t(replicate(ncomb,c(x,y)))  # replicate data
  perms <- combinations(n+m,n,1:(n+m))  # assign to group 1 and 2
  resamp <- rep(0,ncomb)  # to hold the diff in means
  for (i in 1:ncomb)
    resamp[i] <- mean(dta[i,perms[i,]])-mean(dta[i,-perms[i,]])  # calc diff in means
  teststat <- mean(x) - mean(y)  #  observed diff in means
  pvalue <- sum(abs(resamp) >= abs(teststat)) / ncomb   # rel. freq with more extreme diff in means
  list(teststat=teststat, pvalue=pvalue, resamp=resamp)  # return list
}

res <- perm_test (x,y)
res$teststat
res$pvalue
hist(res$resamp, freq=F, main="", xlab="diff. in sample means")
abline(v=res$teststat, col = "red")

A histogram visualising the estimated sample distribution of the difference in sample means looks like this.

The red line indicates the observed difference of 290. We see that the observed difference lies in the right-end tail of the (estimated) sample distribution and is quite unusual if we assume there is no difference between the two groups (i.e., the assignments to the two groups is arbitrary).

The normal procedure is now: The p-value is $0.0086$, i.e. the probability that the difference in sample means deviates (in absolute value) more than 290 from 0. Hence, we conclude that there is a statistically significant difference in mean between the two groups.

Now, you want calculate the probability that the difference in sample means exceeds some number (the standard error (?), if I understood you correctly). We could do this using our sample distribution obtained through resampling. Just calculate the relative frequency of resampled difference in means that exceeds that number (e.g. the standard error $\approx 70.5$), i.e.,

sum(abs(res$resamp) > sqrt(var(x)/length(x)+var(y)/length(y))) / length(res$resamp)

It is approximately 0.55. However, I'm not sure I understand why you want to look at this probability.

Answer 2

Let $\mu_1$, $\mu_2$ be the population means (i.e. true mean heat producing capacity) of Mines $1$ and $2$, respectively. Let $\bar x_1 = 8230$, $\bar x_2 = 7940$ be the observed sample means from samples of sizes $n_1 = 5$ and $n_2 = 6$, respectively, from Mines $1$ and $2$. Finally, let $s_1 = 125.5$ and $s_2 = 104.5$ be the observed sample standard deviations of the heat-producing capacity.

The hypothesis to be tested is $$H_0 : \mu_1 = \mu_2 \quad \text{vs.} \quad H_a : \mu_1 \ne \mu_2,$$ and the test statistic we will employ is the Welch's t-test $$T \mid H_0 = \frac{\bar x_1 - \bar x_2}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}} \sim \operatorname{StudentsT}(\nu),$$ where $$\nu \approx \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{s_1^4}{n_1^2(n_1 - 1)} + \frac{s_2^4}{n_2^2(n_2 - 1)}}$$ is the Satterthwaite approximation for the degrees of freedom. The critical value for this test is $t_{\nu, \alpha/2}^*$, the upper $\alpha/2$ quantile for the Student's $t$ distribution with $\nu$ degrees of freedom. If $|T| > t_{\nu, \alpha/2}^*$, then we reject $H_0$ at the $100(1-\alpha)\%$ confidence level and conclude that the true means are unequal. We may also compute a $p$-value for the test; I obtained $$p \approx 0.00350541.$$

The justification for using the Welch test statistic is that the sample variances are not similar in magnitude. The resulting $p$-value is therefore larger than a test based on the usual two independent sample $t$-test.