I am looking at a decision making process related to a large group of agents making decisions on an appeal against a certain type of fine issued to members of the public. So the decision outcome can either be agents accepts the appeal in which case its cancelled or its rejected and the appellant still needs to pay. I have noticed that there is large variability in the acceptance rate for these appeals i.e. some agents accept 90% of appeals while others only 30%. To me this indicates that perhaps the guidelines instructing agents how they should adjudicate these appeals is not clear and leaves too much room for noise in terms of an agents leniency or stinginess or perhaps laziness, as they just decide the case one way or the other without evaluating it. We can assume that the appeals are assigned to the agents at random so there is no reason to think that certain agents are more likely to be assigned appeals that have a higher probability of being valid. So my question is for this type of scenario how do I know what is an acceptable or expected level of variability in terms of acceptance rates and how do I show that the rates I am seeing fall outside this (if in fact they do) and by how much. I should note, when I was gathering the data for this I looked at appeals over a 4 month period and only considered agents who had made 100 or more decisions. Is this enough of a sample?
Any advice would be much appreciated.