Paired T-Tests vs Independent

Question

The effectiveness of a training course is examined, and performance of each individual in a group is taken both before and after, and the differences are used in a paired T test.

Would it be possible to also perform a two-independent-samples t test to investigate the mean difference if the data before and after were mixed as to no longer be paired? Or would the independent condition still not be satisfied, as the two sets of observations are not independent?

Answer 1

There are two reasons not to do a two-sample t test on paired data.

First, while scrambling the observations may obscure some of the dependence inherent in pairing, scrambling does not address the difficulty that you have only one random sample of subjects from the population of interest; not two independent random samples 'treated' in different ways.

Second, generally speaking a paired test on $n$ pairs has better power than a two-sample test on two independent samples, each of size $n$. Pairing helps to control for among-subject variability.

---

Here is an example with realistic (but fake) data, illustrating the power issue.

 x1 = round(rnorm(15, 100, 10))  # beginning scores of 15 subjects
 lrn  = rnorm(15, 5, 2)          # perhaps they learn something
 x2 = round(x1 + lrn)            # corresponding ending scores

 rbind(1:15, x1, x2)
       1    2    3    4    5    6    7    8    9    10    11    12    13    14   15
 x1  120   89  109  115   96   89  103   92   94   103    96   120   104    87   86
 x2  124   96  111  122  103   94  108   95   99   109   101   127   106    93   92

---

Incorrect two-sample test (ignores pairing); nowhere near significant.

 t.test(x1, x2, alte="two")

         Welch Two Sample t-test

 data:  x1 and x2 
 t = -1.2158, df = 28, p-value = 0.2342
 alternative hypothesis: 
    true difference in means is not equal to 0 
 95 percent confidence interval:
  -13.782202   3.515536 
 sample estimates:
 mean of x mean of y 
  100.2000  105.3333 

---

Correct paired t test; highly significant. (Notice that all 15 differences are positive.)

 t.test(x1, x2, pair=T, alte="two")

         Paired t-test

 data:  x1 and x2 
 t = -11.5151, df = 14, p-value = 1.585e-08
 alternative hypothesis: 
    true difference in means is not equal to 0 
 95 percent confidence interval:
   -6.089461 -4.177205 
 sample estimates:
 mean of the differences 
          -5.133333