Given two sets of product ratings, how to test whether one product is better than the other?

Question

My wife and I are considering buying a soundbar. Two different soundbars stood out to us on Amazon. One had ~2000 reviews, at an average of 4.1 stars. The other had ~80 reviews, at an average of 4.3 stars.

What would be the best way to test the hypothesis that the "true mean" rating of the latter (product B) is higher than that of the former (product A)?

Answer 1

If you can download the ratings, you can either do a T test (after calculating the SDs), or you can do a permutation test.

To do the permutation test, you would put the data in the form \begin{pmatrix} Product & Rating \\ A & \text{Rating}_{1A}\\ \dots & \dots\\ A & \text{Rating}_{1N_A} \\ B & \text{Rating}_{1B}\\ \dots & \dots\\ B & \text{Rating}_{1N_B} \end{pmatrix}

Where $N_A$ and $N_B$ are the number of reviews for each product. Basically what you do is

1. Randomly shuffle the product rating column around while keeping the product label in the same spot

2. Calculate the difference in the new means associated with products A and B

3. Repeat 1. and 2. a million times (just do it a lot)

4. The absolute observed difference between the average rating for products A and B is 0.2, so for two sided test, you just calculate the proportion of reshuffled mean differences that have an absolute difference greater than 0.2