I have 50 days of data of conversion rates for my company. For these 50 days, I have two samples: Control and Test. I have manipulated it to look like the below, with each day being an observation (n=50):
Date Control Test
9/1/2020 2.50% 2.67%
9/2/2020 2.85% 2.49%
9/3/2020 2.10% 2.12%
9/4/2020 2.34% 3.00%
...
I want to know if the conversion rate for the Test Sample is statistically different than the conversion for the Control Sample.
I also have the inputs for these conversion rates for each day. That is, for 9/1/2020, I know there were 1,000 observations for the Control and 25 conversions, yielding a 2.50% conversion rate. And for the Test on 9/1/2020, I know that there were 1,500 observations and 40 conversions, yielding a 2.67% conversion rate. I know this information for all 50 days for both samples.
When I look at this problem, I am unsure of whether to use a test of proportion or a test of means. Yes, conversion rate is a proportion, but when I put the data into the above format (with each row having a conversion rate), I feel like I could just as easily perform a test of means.
For example, for the 50 observations, the mean Conversion Rate for the Control is 2.49% and the mean Conversion Rate for the Test is 2.58%. Using these figures and the standard deviation, I can calculate a means t-test to see if the two samples have statistically significant differences in their means.
However, I could just as easily run a Proportions Test, using the inputs into conversion each conversion rate.
For the Test: Observations = 57,034 | Conversions = 1,475, yields 2.59% conversion rate For the Control: Observations = 99,282 | Conversions = 2,489, yield 2.51% conversion rate
My question is how do I know which to use and what I am gaining/ losing by choosing one over the other? When I run these scenarios in R, I do get different answers, and I am struggling to understand why the difference exists and which is the right method.
Ultimately, it seems like you could convert any test of proportions into a test of means if you really wanted to, just taking the average of the proportions, but this can't be right.
Would greatly appreciate any help on this! Thanks in advance!
You can do both tests assuming you have statistical software and should get qualitatively similar results, although not necessarily identical.
That's because due to the large daily sample sizes, the observed conversion rates are approximately normally distributed. However, since the sample sizes for each day are probably different in each group, the assumption of constant standard deviation for the daily conversion rates for each group is most likely not even approximately satisfied. Think weekends vs. weekdays. That could explain the observed discrepancies.
If it's correct to assume that (a) conversion rates did not change from day to day in each group and (b) outcomes are independent, then it's best to use a proportions test for the overall data for each group. Technically speaking, these four numbers (observations and conversions for both groups) make up a sufficient statistic. In that case you can use Fisher's exact test or simply the proportions test.