The title may be a bit confusing because I not exactly sure how to do/explain what I am after.
I have judges rating an object on a score 1-100, where the majority of objects will in the range 85-95 most of the time (90% or so).
3 judges are needed for the system to be "confident" of the 1-100 rating, however 3 judges judging can take a very long time.
To speed up the system, non-judges can also rate the object but their rating is weighted lower than a judge. Judges have a weight of 2500, and all non-members have their own weight based on a 0 to 2500 scale.
The approval/deny cut off is at a score of 90.
I am looking for a method to cut off ratings once a score of at least 90 has not only been reached but its unlikely that the # of judges it will take to reach 3 will not put it under 90. That is, if 1 judge has rated then a relatively low score from 2 judges will not put in under 90, vice versa if 2 judges have rated, a relatively low score from 1 judge will not put in under 90.
As an example, if 2 judges give the object a score of 97 and 1 member with weight 1600 gives it a score of 99, that is a total of 2*97*2500+99*1600, which would trigger the system that it is very unlikely that the 3rd judge will give it a score low enough to make the average dip below 90, therefore rating should end early, and without the 3rd needed judge.
You could simply apply Bayes' rule. Basically, what you want to do is predict how likely it is that the object will be accepted/denied, given enough time (so that 3 judges make their rating), based only on the first judge rating.
So suppose you have the rating of one judge with rating 95. Then you can calculate from all objects (where you already have 3 ratings) the conditional probability $$Pr(AvRating_3<90|r_1=95)=\frac{Pr(r_1=95|AvRating_3<90) Pr(AvRating_3<90)}{Pr(r_1=95)},$$ where $AvRating_3(=1/3[r_1+r_2+r_3])$ is the rating you use when you have the ratings of all 3 judges, and $r_1$ is the first judge rating of the object.
This probability tells you how likely it is that the final rating after 3 judge-ratings will be below 90 (your cutoff), given that the first rating is, say, $r_1=95$. A new rule based on this probability can be something like: if the conditional probability that the final rating is below 90 is small enough (say, below 20%), then you already approve the object (even though you only have 1 rating so far, but you know, statistically, that with enough time it will likely be approved eventually).
But this only works if you already have many objects with 3 ratings, so that you can calculate these conditional probabilities for all values of $r_1$. For example, if there is no object yet where the first rating was $r_1=95$, then you cannot calculate the above probability. In that case, as an approximation you could average over the immediate neighbors, e.g., $$Pr(AvRating_3<90|r_1=95)=\frac{1/3[Pr(r_1=94|AvRating_3<90)+Pr(r_1=95|AvRating_3<90)+Pr(r_1=96|AvRating_3<90)] Pr(AvRating_3<90)}{1/3[Pr(r_1=94)+Pr(r_1=95)+Pr(r_1=96)]},$$ and if, say, there is no object with $r_1=95$ in your data, you instead use $$Pr(AvRating_3<90|r_1=95)=\frac{1/2[Pr(r_1=94|AvRating_3<90)+Pr(r_1=96|AvRating_3<90)] Pr(AvRating_3<90)}{1/2[Pr(r_1=94)+Pr(r_1=96)]}.$$