When to use which multiple testing correction?

127 Views Asked by At

There are a large number multiple testing p-value correction methods. e.g.:

   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)

I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.

Any ideas? How do I decide which multiple testing correction I should apply?


Disclose: I already posted this in data science but even with a bounty no useful answer was given. Maybe here is more appropriate?