Batch distributions between annotators and supervisors

24 Views Asked by At

5 annotators are asked to annotate 1000 documents. (200 each) Then, 3 of them are chosen to review the 1000 annotated documents. How to organize the distribution so that a supervisor does not review his own annotations?

My solution is

A, B, C, D, E the annotators

A will review 200 documents from D, 133 documents from B

B will review 200 documents from C, 133 documents from A

C will review 200 documents from E, 67 documents from A, 67 documents from B

I am not sure how to mathematically model this problem, so my questions are:

What is the mathematical domain of this problem in order to model the problem? (combinatorics?)

Is there a formula with a variable number of annotators, documents, supervisors, etc...?