Honest vs. Dishonest Voting: does it matter?

Question

Let's say I have a website where people can rate movies on a scale of 0-10. We say people vote honestly when the rating they give to a movie is what they actually think. People vote dishonestly when they try to get the website's rating to be closer to their own rating: for example, if a movie's current average rating is 7.2, but I believe it deserves an 8, I will rate it a 10 to bump the average higher. (Everyone who votes dishonestly always gives a 0 or a 10.)

It's clear that if everyone votes honestly, the website's rating will reflect the true average viewer rating of the movie. My question is: if everyone votes dishonestly, will the website's rating still reflect the true average viewer rating? In other words, does it matter?

Hypothesis: It doesn't matter. Presumably the order of who votes when affects things (when everyone votes dishonestly), but when the number of voters $N$ gets large enough, it should converge.

Follow up questions:

• If it does make a difference, why?

• If it doesn't make a difference, how large of an $N$ do we need before we start to see convergence with the true average? Is that only true if the true ratings follow a certain distribution (uniform vs. normal vs. two-humped, etc.)?

• What if it's actually a mix: some people vote honestly, and some vote dishonestly?

Answer 1

Here's an example that dishonest voting can lead to a result that departs from the true average.

Suppose half the people rate a (bad) movie $0$ and the other half rate it $2$, so the true average rating is $1$. Let's say there are $100$ voters in each camp.

Suppose the $0$-rating camp votes first. Then the first $25$ $2$-raters will give it a dishonest $10$, which brings its average rating up to $(100\cdot0+25\cdot10)/(100+25)=250/125=2$, and after that the rest of the $2$-raters will give it an honest $2$, leaving the rating at $2$, instead of the true average $1$.

This example, of course, is rather artificial, especially in its assumption regarding the order in which votes are cast. But it suggests, to me at least, that dishonest voting is unlikely to come with any guarantee of arriving at the true average. It might be worth asking how close it comes, on some average (such as averaging over the different orderings of the voters), to the true average.

Answer 2

When the mean is closer to one end of the scale, one side of the voting has more to gain than the other from being dishonest, so it's going to bias the voting.

Say that half the voters think that the movie is a 7 out of 10, and half the voters think that the movie is a 9 out of 10. The honest average rating would be 8.

Voters that want the rating to be 7 have great power to influence the rating when it's above 7: they can vote 0 instead of 7. Voters that want the rating to be 9 have very little power to influence the rating when it's below 9: they can vote 10 instead of 9. Because of this, even if the voters are randomly ordered, the long-term average is going to stabilize around 7. (I've tried this out in simulations.)

Here's why. There are three situations the average can be in: it can be in the ranges $[0,7]$, $[7,9]$, and $[9,10]$.

• In the first range $[0,7]$, all voters vote $10$. The average vote is $10$, so the vote trends up to $7$, possibly moving to the second range.
• In the second range $[7,9]$, half the voters vote $0$ and half the voters vote $10$. The average vote is $5$, so the vote trends down to $7$, possibly moving to the first range.
• In the third range $[9,10]$, all the voters vote $0$. The average vote is $0$, so the vote trends down to $9$, moving to the second range.

Eventually, we're going to be alternating between the first and second ranges, and the vote will hover around $7$.

---

In general, this argument tells us that under dishonest voting, the long-term average rating occurs where the average value of a dishonest vote equals the current average. This happens at the intersection of the blue and orange curves in the graphs below; the blue curve (the average value of a dishonest vote) is derived from the ratings of the users, and the orange curve is just the line $y=x$.

(In these graphs, I'm switching to a $1$-to-$10$ rating system rather than $0$-to-$10$ to match real data.)

The first image is the example I gave above. The blue and orange curves intersect when $x=7$.

The second image is based on data taken from https://www.imdb.com/title/tt7286456/ratings. The blue and orange curves intersect when $x=8$. Note that the actual mean is $8.8$ and median is $9$, but under dishonest voting, the few voters that think that the rating should be $8$ or less have more of an impact.