Designing a game for peer evaluation

Question

Say you have a class of $100$ students and you would like the students to grade each other's work. Is it possible to design a game theoretic scheme in which a rational student would mark fairly?

The problems one would want to avoid are:

• Partiality: Students may be tempted to give each other marks based on criteria other than the quality of the work.
• Grade inflation: Students may be tempted to reciprocally give each other unfairly high grades.
• Grade deflation: Students may be tempted to give each other unfairly low grades in hopes of rescaling to "bump" their own grade up. This temptation could drive all evaluations down unfairly.
• Narrow spread: Students may be tempted to mark conservatively, assigning similar marks to the best and worst projects. Low quality reports would unfairly benefit and high quality reports would unfairly suffer.
• Carelessness: Fair marking requires careful evaluation and consideration. Students may be tempted to not put much effort into their evaluations, introducing random noise into the grades and reducing their fidelity.

Out of these problems, I anticipate Grade inflation and Narrow spread are the most serious, followed by Partiality. I am tempted by methods that involve giving a student a fixed number of marks to hand out but that doesn't help the Narrow spread problem at all.

Answer 1

One way to incentivize fairness is to have students grade the papers, and then you grade some papers. If the student gives a score of $n$ and your score is $k \ne n$, the student loses $|n - k|$ points on his own paper. (You could also set a threshold, where a 2 or 3 point difference doesn't lose any points, but that probably makes for grade inflation in general.) Grab papers to grade randomly for each assignment, so there's an incentive to be fair even if your grading got checked on the previous assignment.

To avoid grade deflation, announce that a score of 80-90 is a B, 90-100 is an A, and so on. Then spend time crafting assignments where the relevant performance gets the appropriate grade.

Of course, all of these involve work by you, the instructor. Then again you (like me) are the one getting paid. :)

Answer 2

I would propose the following. It also involves work from you, but not grading:

You start by producing a bunch of copies, say 2 or 3 for each possible grade (with some skipping if it's too much, the importance thing is that the scale should be completely represented). Of course, they have to look like regular copies. Then each student corrects a sample of copies, both real and fake. The grades they got are then altered, depending on how close they got to the supposed mark on the fake copies they corrected. you can give bonuses if they are close and/or maluses if they are far.