I am comparing treatments that select members to receive some email, and we are measuring the response to that email. The emails are similar, but are sent to a different group of members and a different number of members each time. We have priors for the variance in response rate as well as the variance in count per sample.
Because we are splitting at the email level (as opposed to a different variant for each member of a single email), I believe the proper significance test would be based on sigma = the standard deviation of the total response rate per email, and mu1 and mu2 set to the average response rate of emails generated by treatment (or algorithm) 1 and 2 respectively.
Is there any way to calculate the necessary sample size (given a normal type 1 error rate and power) in terms of total number of recipients given the mean and standard deviation of recipients per email, total emails sent per treatment group, and response rate per treatment group?
For example, I can use a standard sample size calculator to calculate the number of samples I need for each treatment when each sample is a response rate (success1/sends1, success2/sends2 ...) for both treatment1 and treatment2, but what if what I have available is sum(all successes) and sum(all sends) for treatment 1 and 2, when I know the mean and standard deviation of the success rate and the send size, assuming both are normally distributed.