My understanding of the binomial distribution is that it can be used to model a random process only if we assume each trial of the process has the same chance of producing a successful outcome, e.g. flipping the same coin N times using the exact same form for each flip (call this Situation 1)
What happens when you have a process that doesn’t have homogenous p, eg Suppose there’s another random process that determines whether you have coin A or coin B (with a = P(coin a is chosen for the trial) and 1-a = b = P(choosing coin b for the trial), and you go on to do N coin-flip trials (call this Situation 2). Would the binomial model still hold, i.e. will the variance of the mean of k-flip experiments in this experiment be p*(1-p), where p = aP(success|coin A is chosen) + bP(success/coin B is chosen)?
I’m asking because I’m wondering if the binomial model is valid for estimating variance in an A/B experiment where participants are given a random treatment and we seek to calculate p-values given observed binary outcomes of the participants ... participants will never have the same initial propensity to have a successful outcome, so I’m likening the real-world scenario to situation 2 described above
Thank you
In general, you cannot apply the binomial distribution if the probability of success ($p$) is varying amongst the trials. That's because you do not have a $1=(p+q)^n$ to develop
However, in case that you propose, if the mechanism is such as to give for each trial a constant probability $s_A$ that the "coin" comes from the set A, with success prob. of $p_A$ and similarly for set B, and clearly $s_A+s_B=1$,
then you can think of performing your trials with a coin that has a balanced mixture of the probabilities $$ p = s_{\,A} p_{\,A} + s_{\,B} p_{\,B} $$