How to calculate survey bias due to preference for first answer?

67 Views Asked by At

I was recently given the results to a survey in which participants chose answers to questions they would be likely to randomly answer, and in which the survey population is known to have a preference for the first answer. Survey takers participated because of a free prize, and now I need to figure out how to account for the bias introduced by the preference for the first answer. How would I go about doing this mathematically?

Here is an equivalent example:

Favorite Representatives Survey

Pick your favorite Utah representative:

  1. Jason Chaffetz (2080 selected)
  2. Rob Bishop (1380 selected)
  3. Chris Stewart (580 selected)
  4. Mia Love (746 selected)

Pick your favorite Kansas representative:

  1. Lynn Jenkins (1890 selected)
  2. Tim Huelskamp (2910 selected)
  3. Mike Pompeo (540 selected)

Pick your favorite New York representative:

  1. Lee Zeldin (2584 selected)
  2. Peter King (1123 selected)
  3. Steve Israel (1790 selected)
  4. Kathleen Rice (1521 selected)
  5. Gregory Meeks (1207 selected)

Pick your favorite Alaska representative:

  1. Donald Young (4585 selected)

Pick your favorite Hawaii representative:

  1. Mark Takai (1900 selected)
  2. Tulsi Gabbard (2700 selected)

As you can see from the data there is a pretty clear trend toward first-choices, but at the same time there are some cases where higher-numbered choices won. What general methods can I use to quantify the effect of the first-choice advantage, and try to find what the results would have been if not for that bias?