I analyse 4 algorithms and 3 sets of metrics for each algorithm in which I apply the non-parametric Kruskal-Wallis test for each metric to detect any differences in performance between these algorithms.
I would like to know whether there is a way to calculate the effect size when applying the Kruskal-Wallis test.
As mentioned in other posts in CV, a post-hoc analysis for Kruskal-Wallis should use the Dunn's test and not the Mann-Witney test for pairwise comparisons between groups (algorithms).
By applying the "inaccurate" MW test, I can calculate the effect size, but what can I do if I apply Dunn's test?
Thanks in advance for any comment/advice.
PS: I posted this question to CV some time ago, but I didn't receive any reply yet. Hence, I post it in this forum too.
First, Dunn-Bonferroni tests are to be done only if the Kruskal-Wallis test indicates there are some differences among the groups. Let's suppose you are testing at the 5% level. Once we know there is some pattern of differences discuss, we can try to discover what that pattern is by making pairwise comparisons among the $g$ treatment groups.
There are $c = C(g, 2) = g(g-1)/2$ possible paired comparisons. If we make each of these at the 5% level, there is a possibility that the 'grand' or 'family' error probability for a pattern of differences emerging from the paired comparisons will be substantially more than 5%.
A Bonferroni procedure is based on the Bonferroni inequality of probability theory. In your application, the idea is that if we test all $c$ comparisons at the level $.05/c$, then the family error probability of the pattern cannot exceed 5%.
So if you have $g = 4$ groups, then $c = 6$ and you should make each multiple comparison at level $.05/6 = 0.0083.$ You could use six 2-sample Wilcoxon tests at that level. or you could look at six Wilcoxon confidence intervals for differences in medians at confidence level $1 - 0.0083$ or 99.2%.
There is no guarantee that the pattern will be absolutely clear. For example, if we have three levels 1, 2, and 3 in increasing order of sample medians, it is possible you might find a clear difference between extremes 1 and 3, but not be able to resolve whether 2 is significantly different from either 1 or 3.
Caveats: I wonder what you mean that you have 4 algorithms and 3 sets of metrics. From that description, I have no idea what your experimental design looks like. Are you doing three separate Kruskal-Wallis tests, one for each 'set of metrics'? If so, what I have said above is OK, with $g = 4.$
Or do you have a two factor design, in which one factor is 'algorithm' and the other is 'metric'? In that case I don't see how a Kruskal-Wallis test can give an appropriate analysis.
If you want to say what the the factors in your design are and how many replications for each factor (or combination thereof), maybe my advice would be different.
Also, I'm wondering what kind of data you have that causes you to think in terms of nonparametric tests.