Total conversions vs individual feature conversion

Question

I have a new website and an old website with 5 features. All features have been updated in the new website.

The old checkout page has a 10% conversion rate. The new checkout page has a 9% conversion rate.

If the user clicks on feature A, On the old checkout page, conversion rate is 23% On the new checkout page, conversion rate is 18%

If a user clicks on feature A, all other parts of the checkout page are still involved. Think about it as a popup that gives additional information, so the conversion rate after clicking on the popup is still affected by other features on the checkout page.

Is it possible to calculate if feature A is better on the new checkout page vs the old checkout page? Is it as simple as yes because the difference is just 5% when the feature is clicked vs 10% in all scenarios?

I do not have any additional information.

Thanks

Answer 1

This is standard A/B testing. Since you do not have a version of the new website with ONLY feature A changed (and the rest as in the old website), you cannot make any conclusions about the impact of changing only feature A. It is possible that A got better but that there are some interaction effects with the other changes that reduces its performance as a whole.