Statistical method of determining "better option" out of the two

Question

Suppose that we are given a work to do and we have two possible options $\text{Option}\; 1, \text{Option}\;2$.

It's reasonable to assume that each option has a fixed success rate $p_i$ for the work.

Now we are given a statistical observations which tell us the following: $n_i$, number of attempts made using $\text{Option}\; i$, and observed success rate of these attempts, $\overline{p_i}$.

Given above, is there a statistical (mathematical) method of determining which option is better?

---

For example, I've been looking at this particular case:

$$n_1=1,374,792\;\;\;\;\;\; \overline{p_1}=\frac{9,483}{1,374,792}\approx 0.0069$$

$$n_2=90,833\;\;\;\;\;\;\overline{p_2}=\frac{819}{90,833}\approx 0.009$$

Intuitively, it is a lot safer (due to WLLN or SLLN) to assume that $p_1=\overline{p_1}$ than to say $p_2=\overline{p_2}$. If we believe that these observations represent true $p_i$ reasonably well, then in this case option 2 is better.

Can we conclude this (or otherwise) in more systematic way using statistical tools?

Any suggestions or reference to relevant information is welcome.

Answer 1

You have plenty of data to conclude beyond a reasonable doubt that the two sample proportions arise from different population probabilities.

A standard test of two binomial proportions found in many elementary statistics texts, uses a normal approximation, which should be very accurate for such large sample sizes. A printout from Minitab's implementation of this test is shown below.

 Test and CI for Two Proportions 

 Sample     X        N  Sample p
 1       9483  1374792  0.006898
 2        819    90833  0.009017

 Difference = p (1) - p (2)
 Estimate for difference:  -0.00211878
 95% CI for difference:  (-0.00274888, -0.00148868)
 Test for difference = 0 (vs ≠ 0):  Z = -6.59  P-Value = 0.000

 Fisher’s exact test: P-Value = 0.000

Notice that the 95% confidence interval does not contain $0$. One might reject the hypothesis that population probabilities are equal if the P-value is less than 0.05 or 0.01, but here the P-value is less than 0.0005. [For a fuller discussion you might see.]

'Fisher's Exact test', for which results are also shown in the printout above, uses the hypergeometric distribution. (It does not rely on a normal approximation.) This test is also explained in elementary statistics texts, and you can google it online. [History and details are given in a Wikipedia article.]

In your case, a more elementary approach seems good enough: You could compare two individual 99% confidence intervals, one for each of the population probabilities $p_i.$

Such a 99% CI is of the form $\hat p \pm 2.576\sqrt{\hat p(1-\hat p)/n}.$ The two intervals are $(0.006716, 0.007080)$ for $p_1$ and $(0.008209, 0.009825)$ for $p_2.$ Notice that these intervals do not come anywhere near overlapping. There are more precise styles of binomial CIs, but your $n$'s are so large that using more precise methods would not make a noticeable difference. These CIs use normal approximations, which should be excellent.

Note: You mention WLLN and SLLN in your question. Both of these important theorems are proved by methods that do not readily lend themselves to accurate numerical comparisons. Except for Fisher's exact test, the methods shown here use the Central Limit Theorem (CLT), which does yield useful (even if approximate) numerical comparisons.