How to generate wrong numerical answers for a multiple choice test without giving information about the correct answer?

Question

I want to generate $n-1$ wrong answers $a_2,\ldots,a_n$ for a multiple choice test given a correct number $a$. The problem is that I don't want the user to be able to guess the correct option with an accuracy of more than $\frac{1}{n}$ if the user doesn't know the correct number. Is that possible and if yes, how?

Ideas

Always draw from the same interval

Ignoring the correct value $a$, just always choose $a_i$ randomly from $[$lower bound, upper bound$]$, e.g. [0,10000]. The problem is that if I ask for e.g. the year of the french revolution ($a=1789$), the user knows that it cannot be e.g. in year 2345. Even if I set the upper bound to the current year, giving the years 110, 720 and 2001 as wrong options will be too easy. Even if the answer isn't a year, a smaller value will be more probable in general.

Draw numbers a bit smaller, larger or around the correct number

If I generate slightly smaller numbers (lets say from the interval $[0.9 \cdot a,a)$) than the correct answer, the user just always chooses the largest one. If I always generate slightly larger ones, $(a,1.1\cdot a]$, the smallest one is always correct. And if I generate numbers around the number, $[0.9 \cdot a,1.1 \cdot a]$, the one in the middle will be more often correct than the others.

Two Step Process

I could first choose a method randomly, 1/3rd of the time I choose smaller, 1/3rd of the time larger and 1/3rd of the time "around" the number. Now I can still break this because if I see that $\frac{\max(a_i)}{min(a_i)} > 1.1$, then I know the last option was chosen and I can choose the middle one. If I change the "around" interval to $[0.95\cdot a,1.05\cdot a]$, the quotient could still be slightly over 1.1. I could use $[0.95\cdot a,0.95\cdot1.1 = 1.045 \cdot a]$ but now $a$ isn't in the center of the interval any more which can be exploited with a uniform draw of random numbers.

Is there any way to create random numbers similar to the correct numbers that don't give any information about it? Maybe I need to use a random number generator that doesn't have a uniform probability in the interval? Or should this exploitability just be factored into the scoring?

P.S.: I guess $0.9\cdot a$ should be $\frac{a}{1.1}$ so that the quotient is the same.

Answer 1

Here are some thoughts about an abstract version of your question.

Assume a finite set of $A$ of possible answers (your questoin suggests that will be an interval of integers). Randomly choose $n-1$ elements from the set of wrong answers. Then provide as choices the correct answer and your wrong answers (place the correct answer at random).

If the test taker knows the answer they will choose it. If they have no information they will guess and succeed with probability $1/n$. If they have some information about the correct answer then they will be able to increase their probability of guessing correctly to some value greater than $1/n$. You can't avoid that unless you know more about how they are thinking.

If you knew their personal probability distribution over the set $A$ of possible answers (something between uniform and certainty) I think you could design a random set of wrong answers that provided minimal information in some sense. Quantifying that strikes me as an interesting and probably difficult problem. When they are certain and correct nothing you do will fool them. If they are certain and incorrect in a way you know should you allow their wrong choice to be one you include along with the correct answer? What would your best strategy be if they thought two of the possible answers were equally likely to be correct?

I suspect this is the kind of game cryptanalysts play,

Answer 2

Let me try to think of this as a classification problem. The student has a set of options and needs to decide whether the option is a correct answer or not. A probabilistic view is also possible, where the student gives odds on each of the answers.

Each student has different abilities to perform the classification. On one end of the spectrum, student knows the correct answer even before looking at the options. On the other end, student thinks all answers are equally likely, because the student knows nothing to help choose.

Considering the French Revolution question, I can imagine that students may progress on the following ability ladder between the two extremes.

1. I know it happened some time in the past
2. There was this bare chested painting of woman on barricade, and lots of beheadings later. No clue when, though.
3. I know it had something to do with the US Independence war, but I don't know which was first
4. US Independence war was first
5. I faintly remember a collection of dates which I memorized for the test, but I don't know which is which.

This will be different for each student, though, so different options are differently likely for each.

In education, there is item response theory, which tries to estimate a model of student's ability by presenting large set of students with large set of questions, and mutually calibrating the questions on students and students on questions. Various models of various complexity can be used to quantify student abilities.

I believe that the question cannot be answered without actual experimentation of some kind (or actual teachers experience substituting for it). If I had the data, I don't know for certain what I'd do with it, but I believe something can be done then.