Should We Use a Paired or Two-Sample Test?

Question

Suppose I have a random sample of size 200. Each sample has two observations about wage in 2000 and wage in 2005, called $w_{00}$ and $w_{05}$ I want to test whether $\mu_{2000} = \mu_{2005}$. I can do it in two ways:

The first way is to carry the usual hypothesis testing on equality about two means, estimating sample mean and variance, and the degree of freedom is 398.

The second way is to transform it into a test on equality about one mean. Specifically I'm testing whether $E(w_{00} - w_{05}) = 0$. Then I would just take the sample mean and divide it by the standard error, and the deg of freedom is 185.

What's the difference between these two methods and which one is more appropriate?

Answer 1

The main issue in your question is the distinction between a 'paired' t test and a 'two-sample' t test of data $X_1$ and $X_2$. (I have changed the title of your question accordingly.)

If you have two (possibly correlated) salary values on each individual, one from the first year and the other from the second year, then you should use the paired test:

Let $D_i = X_{1i} - X_{2i},$ for $i = 1, \dots, n$ and use the test statistic $T = \frac{\bar D}{S_D/\sqrt{n}},$ where $T \sim \mathsf{T}(\nu = n-1)$ under $H_0.$

If you have two independent samples (different subjects in different years), then you should use a two-sample t test. (The Welch 'separate variances' test is generally preferred over the 'pooled variances' test, unless you have strong information in advance that two populations have equal variances. However, with $n$'s as large as 200 there will be little difference between these two versions of the two-sample test.)

Technically, both paired and 2-sample tests require normal data in order to use Student's t distribution to compute critical values or P-values. However both kinds of t tests are reasonably accurate unless the data are markedly skewed with relatively many outliers. With salary data across a broad population, you should check the raw data for normality.

The distinction is of considerable practical importance. Using a 2-sample test for data that are actually paired, can result in failure to detect a real difference.

Example. Here are fake normal data for salaries (in thousands of dollars), simulated according to a paired design with $n = 200$ with a true difference of about 2 (\$2000).

Descriptive statistics:

summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  48.55   69.23   75.74   75.49   82.01   99.50 
## 9.83931    # SD

summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  50.70   70.19   77.44   77.18   84.34  102.80 
## 10.44153   # SD

cor(x1, x2)
## 0.9637345  # Correlation

Results of Tests (using R statistical software): The correct paired test finds a highly significant difference (P-value < .001), but the incorrect 2-sample test does not (P-value of the Welch test about 10%, shown; P-value of the pooled test about the same, not shown.).

t.test(x1, x2, pair=T)

        Paired t-test

data:  x1 and x2 
t = -8.566, df = 199, p-value = 2.936e-15
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -2.082980 -1.303405 
sample estimates:
mean of the differences 
              -1.693193 

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2 
t = -1.669, df = 396.604, p-value = 0.0959
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -3.6876406  0.3012552 
sample estimates:
mean of x  mean of y 
 75.48772   77.18091