Some time back I read this article which addresses various methods of scoring average user feedback and sorting them, and why they don't work:
- Score = (positive feedback) - (negative feedback). Suppose you're sorting definitions on Urban Dictionary by highest to lowest score. Item 1 has 600 positive ratings and 400 negative ratings, so it'll be rated as 60% positive. Item 2 has 5,500 positive and 4,500 negative ratings; it'll be rated at 55% positive, lower than item 1, even though more people gave positive feedback.
- Score = average rating = (positive feedback)/(total feedback). Say you're looking for products on Amazon. Item 1 has two positive ratings and no negative ratings, while item 2 has 100 positive ratings and one negative rating. Item 2 will be scored lower, even though the fact that 100 people gave it five stars should count a lot more than the fact that two people gave Item 1 five stars.
The article's proposed solution is the "Lower bound of Wilson score confidence interval for a Bernoulli parameter" – that is to say, we need to balance the proportion of positive ratings with the uncertainty of a small number of observations, or "Given the ratings I have, there is a 95% chance that the 'real' fraction of positive ratings is at least what?" This can be calculated with the formula:
$$\frac{\hat{p}+\frac{z^2_{\alpha/2}}{2n}\pm z_{\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})+\frac{z^2_{\alpha/2}}{4n}}{n}}}{1+\frac{z^2_{\alpha/2}}{n}}$$
where $\hat{p}$ is the observed fraction of positive ratings, $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal distribution, and $n$ is the total number of ratings.
I found the original article, but I'm having trouble understanding it with my basic high-school-level statistics background. Can someone help me understand the math behind this? Also, are there any pitfalls this equation introduces, and are there other formulae out there which address the problem better?