¿How fast can a movie rating stabilize and how can it maintain over time without fluctuations?

Question

A user posted a question on reddit wondering why rates don't budge on rottentomatoes. Specifically, he is using the example of "starwars: the rise of skywalker ".

You can see the full story here.

Now, let's say someone oversimplifies the math and uses the law of large numbers. The idea is that the ratio of upvotes over the total number of votes will tend to the audience score as time passes. And someone might object saying that this is only true if the conditions remain constant but still in real life, there should be some fluctuations, especially at the very beginning.

You can check the number of users and how the rate is fixed at 86% on the first day starting with 6,231 users and up to now with 79,588 users (you can check how it is moving on the way back machine by clicking on each of the days)

The question remains here of how fast the score stabilized passing from $0$ users to 6, 231 and how it doesn't budge over time.

Answer 1

Let's assume that the initial $6,231$ votes can be thought of as a random sample of all votes.

As you well realized, this is an assumption that can be problematic, as the initial votes may not be representative of all future votes. For example, the first people to put out a vote are probably the kinds of people who have a strong opinion about the movie. The first voters probably went to opening day of the movie ... and people going on opening day probably have higher expectations. These are also the people that are more likely to go through the effort of voting in the first place (i.e. we are dealing with a 'self-selected sample'), than other people who see the movie later on. There are also socio-psychological effects: the 'public perception' of a movie may change over time for a variety of reasons. 'A Christmas Story' had only moderate success in theaters, but now we have 24-hour marathons of it.

Anyway, making that simplifying assumption, the margin of error is about $1$%, i.e. there is a $95$% probability that the 'overall' approval vote is within $1$% of the $86$%. So yes, the fact that this number didn't budge between $6231$ and $79,588$ is completely in line with a statistical analysis.

Now, how fast can we expect a score like this to stabilize? It all depends on what you mean by 'stabilize', but I can tell you that with $1,000$ votes the Margin of Error is already down to about $3$% ... so I would say you can expect it to be pretty stable from that point on. With $100$ votes, the Margin of error is about $10$%, so I wouldn't call that quite 'stable' yet. But yes, on the order of $1,000$ votes is where it's pretty 'stable' ... which is exactly why so many polls ask $1,000$ to $2,000$ people: the expected gain in reliability of the obtained number isn't worth the extra effort of polling, say $10,000$ people, let alone $1,000,000$

Here is a Margin or Error calculator I used

Answer 2

Suppose that in reality 86% of the people would like the film if they saw it. If you take a sample of watchers and ask their opinion, the most likely percentage of people who like it in the sample is 86%. There is some random variation on this, but the larger the sample size, the closer you get to the real percentage.

It is well known, how to handle these things mathematically. The percentage you get from your random sample is normally distributed random variable whose expected value is the actual percentage of the whole population (say 0.86 for 86%), and standard deviation is $$\sqrt{\frac{0.86\cdot 0.14}{n}}$$ where $n$ is the sample size.

From these you can calculate for example the confidence intervals. From your numbers, ($n=6231$), you'd get sd=0.004395763, and the 95%-confidence interval would be from 85.14% to 86.86%. That means that from the small sample the whole population's opinion is within these percentages with the probability of 95%.

So all in all, the numbers don't give any reason to suspect that there's something wrong per se. A sample of 6000 persons would be a solid base for a election survey, and adding even 10000 voters to that doesn't make much difference.

The numbers of course don't say that there's nothing wrong with Rotten Tomatoes either. I think it's a bit suspicious, since opinions change with time. Perhaps the opening weekend had only true fans as audience, and by not the spouses who were forcefully dragged there should make the percentage go down. On the other hand there's tons of other factors too. To make any inference you should have similar information on other movies too.