Estimating grader bias/variance and MLE test scores given multiple graders assigned to grade each test

Question

Suppose we have $m$ graders and $n$ students, and we want to grade a test so that $k$ graders are assigned to grade to each test, and all graders grade the same number of tests. (I realize $m,n,k$ have to satisfy certain properties to make this "perfect assignment" possible, but I'd rather just skip this point and assume it's true). Also, to make things interesting, let's assume the assignment of graders to tests is random (so it's not like a group of graders all have the same set of tests to grade).

Furthermore, let's assume that each grader $i$ has a mean bias $\mu_i$ and a variance $\sigma_i^2$ for that bias associated with their grading, and the bias they apply to each test they grade is sampled independently from a normal distribution with these parameters. And each test $j$ has a "true grade" $c_j$. So then if grader $i$ is assigned to grade test $j$, then the grade they assign will be $c_j + x_{ij}$ where $x_{ij}$ is the sampled bias from the normal distribution with parameters $\mu_i$ and $\sigma_i^2$.

If the $\mu_i$ and $\sigma_i^2$ are unknown, how do we find the maximum likelihood values for the true grade scores $c_j$? If using a prior for graders' parameters is required I guess I'm ok with that. I would also like to know the MLE (or MAP if we go Bayesian) values for the grader parameters $\mu_i$ and $\sigma_i^2$. The idea being that graders with lower estimated variance should be preferred to those with higher variance, if we want as accurate of assigned grades as possible in the future.

I've phrased this in terms of test grading for clairty, but it's actually for an "active learning" problem in machine learning that we are very interested in in our lab, hence insights on this problem could really help.

Answer 1

I'm looking at your first paragraph. Depending on values of m, n, and k, you might make grader assignments according to a balanced incomplete block design or a partially balanced incomplete block design. Without some kind of balance it does not seem possible to disentangle so many unknown means and variances. There is rich literature on such issues of balance. Even if not directly applicable to the situation you really care about, you might get some ideas how to proceed.

A Bayesian approach with a Gibbs Sampler might be useful. But there are so many latent variables, that I wonder whether conclusions might be guided by priors (even noninformative ones) to a greater extent than you would prefer. Again here, some sort of balance may be required.

In either case, it seems that a totally random assignment of tests to graders is not a good design choice.

Answer 2

Ok, here's what I have so far:

For the MLE: You're right. The MLE formula will not have a closed form solution. But, the outline of the formula seems simple enough. Treating each test score as an observation, we have $T_{ij} = c_i + x_{ij}$ ($i$ refers to students and $j$ markers)

Now lets make some assumptions. Suppose that the "true" test scores are also normally distributed with mean $y$ and variance $\Sigma$. Now, each test score is the sum of independent normally distributed variables. Thus, we have that: $TS_{ij}$~$N(y+\mu_j,\Sigma+\sigma_j^2)$

If we knew the values of the types, then the likelihood would be easily given by $L(TS_{ij} | \mu_j,\sigma_j) = \frac{1}{\sqrt{\Sigma+\sigma_j^2}}\phi(\frac{TS_{ij}-y-\mu_j}{\sqrt{\Sigma+\sigma_j^2}})$ where $\phi$ is the standard normal pdf. But, since we do not know each persons type, they need to be integrated out of the likelihood.

$L_{ij}=\int \frac{1}{\sqrt{\Sigma+\sigma_j^2}}\phi(\frac{TS_{ij}-y-\mu_j}{\sqrt{\Sigma+\sigma_j^2}})dF(\mu_j,\sigma_j)$

In terms of identification. I focus on the means only.

Since each grader is assigned tests at random, then on average the average grader's test score will be $y + \mu_j$ -- the average child's test score plus the grader's average bias. There will be $M$ such equations. Similarly, the average of the same test $j$ for the $k$ graders will be $y_j + E[\mu_i]$ -- the child's average test score plus the average marker bias. There are $N$ such equations here. (Note: For simplicity I've imposed the Law the Large numbers here)

This suggests you might have an identification problem. You have $M+N$ equations with $M+N+2$ unknowns. This would seem to suggest some normalizations will be required. For example, you can normalize the average test score and bias to be $0$. $y=E[u_j]=0$