I am currently in the process of creating a game consisting of a fixed set of tasks of varying difficulty. Each player gets the same set of tasks to choose from and is awarded a certain number of points once they solved it. They do not have to commit to the tasks, i.e., they can switch at any time if they want to. There is a time limit on the overall game.
The problem is: I do not feel capable of accurately determining the "real" difficulty of the tasks, so I would like the scoring to dynamically adapt to the number of solves of each task. If a task is solved by many players, it is likely to be rather easy and should be awarded fewer points than one with very few solves.
Of course there are many possible ways to design such a system, but I feel that anything I can think of has some blatant problems. Is there anything comparable to, e.g., the Elo system used for chess, but in the scenario described above instead of symmetric two-party-games? I would like to use a mathematically reasonable scoring system if possible. Has something like this been studied before? I am grateful for any pointers.
Here's the simplest thing I can think of:
Suppose you have $K>1$ tasks. A task $k$ was attempted $N_k^t$ times at time $t$ and successfully solved $n_k^t\le N_k^t$ times at time $t$. The "success rate" of task $K$ is therefore $s_K^t:=n_k^t/N_k^t$ at time $t$.
Now the simplest ranking would be to order these $s_K^t$; smaller rank for higher success rate (easier task). That is, for $K=2$, rank 1 for $\arg\max_K s_K^t$ and rank 2 for $\arg\min_K s_K^t$.
For $K>2$, the rank of task $K$ is $r_K^t=\{\#k\backslash K: s_k^t>s_K^t\}+1$, i.e., the number of other tasks that have a higher success rate than task $K$ plus one. This allows for ties in the success rate, which results in ties for the rank. You can then assign points for the successful completion of the tasks based on the ranks, i.e., fewer points for lower (smaller) ranks.
This procedure dynamically updates the ranks of the tasks as $s_k^t$ develops and therefore also the points gained for them once you assign points by rank. Of course, you could also assign points based on the (cardinal) success rate $s_k^t$ rather than the (ordinal) ranks.