I am comparing the difference of medians between two groups (group 1 and 2) of sample sizes $n1$ and $n2$ that have respective finite populations of size $N1$ and $N2$. Each sample has a respected set of responses $r1$ and $r2$ My null and alternative hypotheses are as follows:
$H_0$: $med(r1) \neq med(r2)$
$H_a$: $med(r1) = med(r2)$
I would like to confirm that my boostrap approach for finite population size without pooling sample data correctly provides a distribution function of the differences between samples.
Assuming a large population, I can compute the distribution of medians for each group using bootstrapping with replacement. To check if the observed difference is due to random error, use the following approach:
Approach 1, assume large population
- pool the sample responses from two groups together into a list of length $n1 + n2$,
- shuffle the pool,
- split the pool into "simulated" groups--cutting the shuffled list into new lists of responses of sizes $n1$ and $n2$,
- compute the medians of each simulated pool,
- compute the differences of the medians in each pool,
- repeat steps 2-5 many times to calculate a set of medians, and
- use the resulting cumulative distribution function of set of medians to understand the probability of observing various effect sizes due to chance (i.e., bin and count the results, divide the counts by the total number of resamples). A similar example of this approach is in A.B. Downey's Think Stats (pg 105).
Now, for a finite population size, A.C. Davidson and D.V. Hinkley's "bootstrap methods and their applications" provide methods modify sample size when bootstrapping statistics estimating a population quantity, where the population is a known, finite size (pg 92). For example, given a finite population size, we can adjust the resample size upwards to $n'$ where $n'=(n-1)/(1-n/N)$. Here, $N$ is the population size.
I think that my above steps for shuffling a pool break down, because I'm now working with an $n1'$ and $n2'$ sample size. So I went with the following approach:
Approach 2, fixed population
- compute $n1'$ and $n2'$ based on fixed populations sizes
- bootstrap the median test statistics for group 1 and group 2 many times while resampling with replacement from the pool of responses many times. That is, resample with replacement from all group 1 and group 2 responses to create simulated groups of size $n1'$ and $n2'$
- calculate the difference in medians between the groups (calculated in step 2)
- use the cumulative distribution function of the resulting differences to explore probabilities of observing given differences between the medians.
Is approach 2 correct? The difference to me seems that in Approach 1, I reshuffle the data, effectively redistributing the data without replacement. But in approach 2, I'm redistributing with replacement. If approach 2 is flawed, how do I correct my methodology to find the probability of getting the effect (i.e., differences in medians) due to random sampling error?
A more concrete toy example to better explain the type of problem that I'm trying to solve: suppose I have a school where a teacher lectures to both class 1 and class 2 with respective class sizes of 15 and 20 students. I survey 5 students in class 1 and 10 students in class 2. The responses from class 1 are {1,2,3,4,5}. The responses from class 2 are {2,3,4,5,6,7,2,3,4,5}. I want to know if the attitude between the two classes taught by this teacher are different.