Authoritative measure for rating photos in a photo contest - practical issue

Question

I am trying to find a good algorithm that would serve as an authoritative way to assess pictures provided for a photo contest. There is a bunch of photos that came for the contest. Each person from a group of people gets all of the photos to mark from 1 to 5, but they don't have to mark all of the photos, only the ones they want to. After we have all the results, we would like to tell which of the photos won. The intuitive average won't work that well: imagine a photo that got 100 times a 5 and once a 4 and a photo that got only one 5. The latter would have bigger average though you might think that the first photo got bigger interest and in the end it was marked with 5 by the whole 100 of people. Sum won't work also, as 3 and 3 would value more that 5. What measure could I use to rate the photos in an authoritative way so that we could tell which photo has won?

Answer 1

I am not sure that this question perfectly fits this website and especially "measure-theory" tag. Since it is rather a multidisciplinary question, I guess it is worth to ask it somewhere else, like on User Experience or Game Development, perhaps they are more familiar with such algorithms. Unlikely, but maybe Webmasters or Web Applications or even Statistics would also work.

Based on the problem you've described it seems that more fair would be to introduce sum of shifted grades. The problem of 1-5 grade system is that 1 and 2 are positive numbers which should bring a negative effect. For the grading of photos on the web it also maybe even more biased since it is rare to see there something but 4 or 5.

The idea then is to introduce a 'fair price' $p$ for the lack of attention which is not as bad as $0$ which is even less than $1$, and not as good as, say $4$. Then you shift grades with respect to it while summing. As an example - we can take $p=3$ and then the overall grade for the photo will be $$ G(g_1,\dots,g_n) = (g_1-3)+(g_2-3)+\dots+(g_n-3) $$ where $g_1,\dots,g_n$ are grades gifted by users. Taking your examples into account, in the first case of $100$ fives and single $4$ we have $G = 202$ and for the single five we have $G = 2$ which seems to be fair, since the first photo attracted more attention. For the second example, we have $G(3,3) = 0$ while $G(5) = 2>G(3,3)$ which again seems to be fair.

The problem you can encounter - is that $p=3$ is not a fair price for the lack of attention in the case when almost all grades are $4$ or $5$. I guess it means that $4$ is already 'bad'. To tackle this problem, the simple suggestion is to take the average grade as a value of $p$. Hope to be of help and good luck.