KS test: how to test if dist1 is on average larger than dist2

Question

I've got two numerical vectors d1 and d2. I want to test if values of d2 are larger than d1 values. If I perform two-tailed KS test, I get p2tailed=0.0840 and if I perform one-tailed KS test with the hypothesis that d1 is larger than d2 I get p1islarger=0.0460.

Can I say that the probability to reject the hypothesis that d1 is lower or equal than d2 is p2tailed-p1islarger?

Otherwise, how can I calculate it?

Thanks.

Answer 1

For small p-values, the p-value of a 2-sided test tends to be larger than the p-value for the one-sided test (in the correct direction) for the same data. About double for roughly symmetrical distributions.

Example with fake data, analysis using R statistical software:

 x = round(rnorm(10, 120, 15),2)  # generate x's
 x
 ##  99.98 108.48 120.31 121.97  98.17 133.71 112.28  82.91 117.22 120.44
 summary(x)
 ##   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 ##  82.91  102.10  114.80  111.50  120.40  133.70 

 y = round(rnorm(10, 100, 15),2)
 y
 ##  94.18 112.39 104.15 112.87 115.82  95.81  94.43 111.53 111.55 108.80
 summary(y)
 ##   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 ##  94.18   97.90  110.20  106.20  112.20  115.80

 xy = c(x, y);  gp = as.factor(rep(c("x","y"), each=10))
 boxplot(xy ~ gp, horizontal=T)

 ks.test(x, y, alternative="two.sided")

 ##        Two-sample Kolmogorov-Smirnov test

 ## data:  x and y 
 ## D = 0.5, p-value = 0.1678
 ## alternative hypothesis: two.sided 

 ks.test(x, y, alternative="less")

 ##        Two-sample Kolmogorov-Smirnov test

 ## data:  x and y 
 ##  D^- = 0.5, p-value = 0.08208
 ## alternative hypothesis: less 

In the example above, the p-value changed from about 17% (2-sided) to about 8% (1-sided).

This is why dishonest researchers sometimes make rationalizations to use a one-sided test (after they have seen the data), when a two-sided test is really required. The one-sided test has a smaller p-value--maybe small enough to claim one can reject.