Computing the confidence interval for cohen's d?

Question

When I have the standard error of d (standard mean difference), how I should compute the confidence interval of it? If you know a tutorial please introduce to me.

Answer 1

Background. Suppose you have two independent random samples, say $X_1, \dots, X_n$ from $\mathsf{Norm}(\mu_x, \sigma_x)$ and $Y_1, \dots, Y_m$ from $\mathsf{Norm}(\mu_y, \sigma_y).$ If you test $H_0: \mu_x = \mu_y$ against $H_a: \mu_x \ne \mu_y,$ you should use a Welch separate variances t test. If you are fairly certain that $\sigma_x = \sigma_y,$ you might use the pooled two-sample t test. Recently, use of the pooled test has been deprecated, except when sample sizes are very small or there is substantial prior evidence or experience that $\sigma_x = \sigma_y.$

Effect size. Cohen's d (earlier Hedge's g) is intended to measure 'effect size' in the pooled setting. Effect size is defined as $\theta = \frac{\mu_x - \mu_y}{\sigma},$ where $\sigma = \sigma_x = \sigma_y.$

Commonly, $\mu_x$ is estimated by $\bar X,\,$ $\mu_y$ is estimated by $\bar Y,\,$ and $\sigma$ is estimated by $$S_p = = \sqrt{\frac{(m-1)S_x^2 + (n-1)S_y^2}{m+n-2}},$$ where $S_x$ and $S_y$ are sample standard deviations of the two samples (taken individually). Then $\theta$ is estimated by Cohen's d: $$d = \frac{\bar X - \bar Y}{S_p}.$$

This is the same as Student's pooled t statistic, but the context in which it is used is somewhat different: If $H_0$ is true, then $T = (\bar X - \bar Y)/S_p$ has Student's t distribution with degrees of freedom $\nu = m + n - 2.$ However, effect size is mainly of interest when $H_0$ is false. If $H_0$ is rejected, a traditional approach has been to make a confidence interval (CI) for $\delta = \mu_x - \mu_y$ and the formula for that CI can be found in most elementary statistics texts.

CI for effect size. You want to find a CI for $\theta.$ Unfortunately, when $H_0$ is not true, the distribution of $d$ has a non-central t distribution. The article I linked in my comment explains how to use R statistical software to get a CI for $\theta$ based on $d,$ using commands in one of the R libraries. A program for a parametric bootstrap CI for $\theta$ is illustrated below.

Example. Consider fake data, with a random sample of $n = 10$ values $X_i \sim \mathsf{Norm}(115, 10)$ and an independent random sample of $m = 20$ values $Y_i \sim \mathsf{Norm}(100, 10).$ Thus $H_0$ is false, $\delta = 15$ and $\theta = 15/10 = 1.5.$ Below we show how the data are generated, and do a pooled t test of $H_0$ vs. $H_0.$ It happens that $H_0$ is rejected and that a 95% CI for $\delta$ is $(2.18, 17.38).$ The CI contains $\bar X - \bar Y$ (as it must); it also happens to contain $\delta = 15.$ Also, $S_p = 9.58,$ which not far from $\sigma = 10,$ considering the small sample sizes.

set.seed(318);  n = 10;  m = 20
mu.x = 115;  mu.y = 100;  sg = 10
x = rnorm(n, mu.x, sg);  y = rnorm(m, mu.y, sg)
t.test(x, y, var.eq=T)

        Two Sample t-test

data:  x and y 
t = 2.6352, df = 28, p-value = 0.01355
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
  2.177199 17.377599 
sample estimates:
mean of x mean of y 
 110.4174  100.6400 
s.p = sqrt((9*var(x) + 19*var(y))/ 28);  s.p
## 9.579947

The data are plotted below:

In order to get a CI for $\theta,$ we can use a bootstrap procedure. If we knew the true distribution of Cohen's d in this situation, we could use it to get bounds $L$ and $U$ such that $P(L < d < U) = 1.96,$ so that a 95% CI for $\theta$ would be $(L, U).$

Not knowing this distribution, we use available data $\bar X, \bar Y,$ and $S_p$ as estimates of $\mu_x, \mu_y,$ and $\sigma,$ respectively. Then we 're-sample' $B = 10,000$ values of $d = T,$ and we take quantiles 0.025 and 0.975 of the resulting re-sampled distribution of $d,$ to serve as approximate confidence bounds. The resulting 95% CI for $\theta$ is $(0.634, 5.013),$ which contains the observed value of $d = T = 2.64$ (from the t test above).

Because we simulated the data, we know that the true effect size is $\theta = 1.5$ and the CI happens to contain that value also. [The bootstrap procedure involves simulation as a computational device. For the same data, it will give slightly different results on each run. Use the seed 123 as shown to get exactly the same result; omit the set.seed statement or change the seed for a fresh run. One additional run gave the CI $(0.633, 5.012).]$

set.seed(123); B = 10^4  
n=10; m=20; a.obs=mean(x); b.obs=mean(y); sg.obs=sqrt( (9*var(x)+19*var(y)) /28 )
a.obs;  b.obs;  sg.obs  #values observed in sample (from above)
## 110.4174
## 100.64
## 9.579947
d.re = replicate(B, t.test(rnorm(n,a.obs,sg.obs),rnorm(m,b.obs,sg.obs), var.eq=T)$stat)
L.re = quantile(d.re, .025); U.re = quantile(d.re, .975)
as.numeric(c(L.re,U.re))
## 0.6337554 5.0126049

The figure below shows a histogram of the bootstrapped values of Cohen's d along with the bootstrap confidence bounds (red).

Note: As a trick to simplify the R code for the Bootstrap procedure, we use the built-in function t.test and use only the test statistic $T$ (obtained with $stat) for the re-sampled value d.re of Cohen's d.