Two-sample t-test: What is actually happening?

592 Views Asked by At

I am doing a t-test and I get a number and compare it to my critical value. great, but I actually dont understand what I am doing.

Can you please describe what is actually happening with a t-test? How does it compare when I draw the 2 normal distributions? is it the overlap or something like that?

1

There are 1 best solutions below

9
On BEST ANSWER

Apparently, you are talking about a two-sample t test, which means $\bar X_1$ and $\bar X_2$ of two samples are used to judge whether the means $\mu_1$ and $\mu_2$ (of the two populations from which the samples were drawn) are equal.

The null hypothesis is $H_0: \mu_1 = \mu_2,$ or equivalently $H_0: \mu_1 - \mu_2 = 0,$ and the alternative hypothesis is $H_a: \mu_1 \ne \mu_2,$ or equivalently $H_a: \mu_1 - \mu_2 \ne 0.$

Now, $\bar X_1$ is a good estimate of $\mu_1$ and $\bar X_2$ is a good estimate of $\mu_2.$ Thus, $\bar X_1 - \bar X_2$ is a good estimate of $\mu_2 - \mu_2.$

Accordingly, if $\bar X_1$ is 'far from' $\bar X_2$ we believe that $\mu_1$ may be far from $\mu_2$ and we will reject $H_0.$ But if the two sample means are 'near to' each other, we will not reject $H_0.$

So our task is to find a boundary line between 'near to' and 'far from'. To do this wisely, it is clear that we need information in addition to the two sample means. It is easier to judge 'nearness' if the populations are very tightly concentrated around their means (with small variance). Also it is easier to judge 'nearness' if we have large sample sizes rather than smaller ones.

Because of this, we need a formula to judge 'distance' between the sample means that also involves the sample sizes $n_1$ and $n_2$ and the sample variances $S_1^2$ and $S_2^2.$ One useful formula is $$T^\prime = \frac{\bar X_1 - \bar X_2}{\sqrt{\frac{S_1^2}{n_1}+\frac{S_2^2}{n_2}}}.$$

Notice that small values $S_i^2$ tend to make $T^\prime$ larger, and that large values $n_i$ tend to do the same.

Theory and simulation studies have shown that $T^\prime$ has very nearly Student's t distribution with degrees of freedom $\nu^\prime$ determined by $n_1, n_2, S_1^2,$ and $S_2^2.$ (The value of $\nu^\prime$ is never smaller than the minimum of $n_1-1$ and $n_2 - 1$ and never larger than $n_1 + n_2 - 2.$) So you can look in a printed table of Student's t distribution (or use software) to find a critical value $t^*$ that separates 'near' and 'far', at a given significance level (often 5%). If $|T^\prime| > t^*,$ then reject $H_0$, otherwise do not reject.

In summary, the $T^\prime$ statistic, with $\bar X_1 - \bar X_2$ as its numerator, is a more effective way to judge the distance between the two sample means than to use simply the difference $\bar X_1 - \bar X_2$ itself.

Note: I have shown you the so-called 'Welch' or 'separate-variances' version of the $T$-statistic, which is preferred by most practicing statisticians. Your textbook may show the 'pooled' version, which is a little simpler, but makes the (often unwarranted) assumption that the two populations have the same variance.

Numerical examples added after lunch:

The plot below, from R statistical software, shows population curves for $Norm(\mu_1 = 100, \sigma_1 = 20)$ and $Norm(\mu_2 = 125, \sigma_2 = 30).$ A random sample of size $n_1 = 10$ from the first population gives $\bar X_1 = 92.34$ and $S_1^2 = 396.6$ (tall thin red tick marks); a sample of size $n_2 = 12$ from the second population gives $\bar X_2 = 121.15$ and $S_2^2 = 855.3$ (shorter thicker blue tick marks).

enter image description here

Standard computations give $|T^\prime| = 2.74$ with degrees of freedom $\nu^\prime = 19.$ Thus the critical value is $t^* = 2.09$ and the null hypothesis is rejected at the 5% level.


Persuant to a Comment Q&A: Here is a plot of the density function of Student's t distribution with 19 degrees of freedom. The rejection region is outside the vertical red dashed lines (each of which cuts 2.5% from one tail of this distribution. In this example $T^\prime = -2.74.$ The position of this observed value of the test statistic is plotted as a solid green line. The P-value (much less than 5%) of the test is the total area to the left of the solid green line and the dotted green at +2.74, which is just as far from 0 in the opposite direction.

enter image description here


By contrast, samples of sizes $n_1 = 8$ and $n_2 = 10$ (not plotted) from the same populations give $|T^\prime| = 1.90,$ degrees of freedom $\nu^\prime = 13,$ and critical value $t^* = 2.16,$ so $H_0$ is not rejected at the 5% level. One says that the smaller sample sizes have resulted in a less powerful test, which is unable to distinguish between $\mu_1 = 100$ and $\mu_2 = 125,$ in the circumstances described.

Note: For each example, data for the pair of samples were randomly generated in R beginning with set.seed(1234). This should make it possible for interested visitors to generate the same samples we used.