Multiple testing: allocation of type 1 error

76 Views Asked by At

Suppose I have $K$ independent test statistics $T_i$, with each derived from a sample of size $n_i$. It follows that $\sqrt{n_i} T_i\sim N(0,1)$ under the null.

If one were to use Bonferonni correction for multiple testing to control for family-wise error rate $\alpha$, each test would be allocated to a type I error rate of $\alpha/K$. Bonferonni is known to have a high type-II error. I understand that there are modified Bonferonni such as Holmes or Hochberg procedure. But they seem to be based on the rank of the p-values and may still be not very powerful.

Is there any known procedure (proposed somewhere) under such a scenario that allocates type I error based on relative sample size ${(n_i/n_k)}^{-1/2}$, when $n_i/n_k$ converges in some sense to a constant for any pairs of $i$ and $k$? That is we allocate $\alpha_i=\alpha \frac{n_i^{-1/2}}{\sum_{k=1}^K n_k^{-1/2}}$. How would this be compared to Bonferroni?

If the above is not a sensible procedure, why not?

1

There are 1 best solutions below

0
On BEST ANSWER

Actually Bonferroni's remains true for any sequence $\alpha_1, \cdots, \alpha_K$ satisfying $\sum_{k=1}^{K}\alpha_k = 1$, which your formulation does satisfy. For example, if $p_1, \cdots, p_K$ were the $p$-values, then \begin{align*} \text{FWER} = P\left(\bigcup_{k=1}^{K}(p_k \le \alpha_k)\bigg|H_0\right) \le \sum_{k=1}^{K}P\left(p_k \le \alpha_k|H_0\right) = \sum_{k=1}^{K}\alpha_k = \alpha \end{align*} The more important question is if the choice of $\alpha_i = \alpha \frac{n_i^{-1/2}}{\sum_{k=1}^{K}n_k^{-1/2}}$ is optimal. Minimizing Type II error is equivalent to maximizing power. Our power under such a test would be \begin{align*} \beta &= P\left(\bigcup_{k=1}^{K}(p_k \le \alpha_k)\bigg|H_1\right) \\ &= 1 - P\left(\bigcap_{k=1}^{K}(p_k > \alpha_i)\bigg|H_1\right) \\ &\ge 1 - \bigwedge_{k=1}^{K}P(p_k > \alpha|H_1) \\ &= 1 - \bigwedge_{k=1}^{K}(1 - \beta_k) \\ &= \bigvee_{k=1}^{K}\beta_k \end{align*} where $\wedge$ and $\vee$ mean min and max, respectively. We can't exactly maximize the power, but we can at least maximize the lower bound. What this procedure implies is to simply allocate all of $\alpha$ to one of the $\alpha_k$ corresponding to the singular test with the most power. This is a bit underwhelming, but it guarantees a conversative control for the Type II error. A liberal control would be setting an upper bound for $\beta$, which can be derived from Bonferroni's inequality as \begin{align*} \beta \le \left(\sum_{k=1}^{K} \beta_k\right)\vee 1 \end{align*} So, we then optimize $\alpha_1, \cdots, \alpha_K$ to maximize the sum of the $\beta_k$'s.