Seeking a better skill calculation of a professional Magic: The Gathering player

Question

One of the interesting things about the professional Magic: The Gathering ecosystem is that players who devote their entire lives to being great at the game might have a peak win percentage of 60%-65%. Indeed, the distance between an above average and elite player might be just 5-10%. This seemed like an interesting math challenge, but first a small bit of background:

In tournaments, the rounds are typically structured so that players are paired as evenly as possible. That is, you will likely be playing against someone with a similar W/L record as you. Ignoring skill and variance for now, a very basic interpretation of this pairing system is that you should expect to win around 50% of the time.

Given that, how could one represent the true skill distance between players? For example, just how much better is a 65% player than a 60% player? Based on the unbelievable rarity of even a 70% win rate, perhaps a 65% player would be an order of magnitude better than a 60% player?

So I could imagine a formula in which 50% = 0.5 with that value trending exponentially towards down towards 0 or up towards 1 depending on the win rate.

Thoughts?

Answer 1

Ignoring the MTG aspect and instead concentrating on the ranking system of players with disparte skill levels, consider the Elo ranking system originally designed for chess (now used in Scrabble, Baseball, Football, and e-Sports). Elo relies on logistic distribution to approximate the underlying skill levels of the players. From the wiki:

The difference in the ratings between two players serves as a predictor of the outcome of a match. Two players with equal ratings who play against each other are expected to score an equal number of wins. A player whose rating is 100 points greater than their opponent's is expected to score 64%; if the difference is 200 points, then the expected score for the stronger player is 76%.

It looks like there are several projects already online that attempt to rank players using the elo system, here's one for example:

http://www.mtgeloproject.net/

Another interesting discussion on the parameterization needed for Elo w.r.t. MTG can be found on this thread:

https://www.reddit.com/r/magicTCG/comments/3pxuvp/mtg_elo_ratings_for_top_100_constructed_limited/

These ratings use a parameter K, which determines the responsiveness of ratings to match results. If K is too high, ratings respond too much to match results, and vice versa. This parameter is calibrated separately for constructed and limited matches to match results as closely as possible. For limited the calibrated value of K is 11, and for constructed the calibrated value of K is 8.5.

Answer 2

This answer addresses the problem more abstractly. Imagine the following unrealistic, highly idealized assumptions actually hold:

• Every player $x$ has a true skill score $s(x)$ which is a real number, and higher skill means a better player.

• There is a function $f$ that, given any two scores $s(x), s(y)$, outputs the win prob $f(s(x),s(y))$ for player $x$ when the opponent is $y$. Further it has a bunch of nice monotonicity and symmetry properties e.g.

  • $a \ge b \implies f(a,c) \ge f(b,c)$,

  • $c \le d \implies f(a,c) \ge f(a, d)$,

  • $a \ge b \implies f(a, b) \ge 1/2$, etc.

In this unrealistic, highly idealized case, you might say their skill "distance" is $s(x) - s(y)$ or $45$ points, or that $x$ is better than $y$ by a "ratio" ${s(x) \over s(y)}$ or $127\%$. However, saying these things requires everybody can measure $s()$. If it is actually a hidden variable, then you can redefine some new skill score using any monotonic transformation e.g.

• $s'(x) = 1000 \times s(x)$

• $s'(x) = 1000 \times s(x) + 375$

• $s'(x) = s(x)^2$

• $s'(x) = 10^{s(x)}$

And you can change $f$ appropriately to $f'$ and this new $f'$ would still calculate the win prob and have all the nice monotonicity and symmetry properties you want. However, the "distance" $s'(x) - s'(y)$ or "ratio" ${s'(x) \over s'(y)}$ would now be different numerical values. In other words, even if the player skills never change and you observe an infinite number of games to find out the true win probs for every pair of players, you cannot distinguish between $(s,f)$ vs $(s', f')$ as the "underlying" model, so to speak.

To make the point more simply, a statement like "$x$ is $27\%$ better than $y$" or "$x$ is $45$ points better than $y$" is meaningless unless everyone agrees on a way to measure skill score.

Having said all this, ELO is certainly a common way to go. In effect it is an idealized statistical model that, given the observed win probs, back-calculates $s$ (and also $f$) under a bunch of pre-agreed design choices / conventions (including many "fudge factors" or "parameters" for tuning).

---

BTW, IMHO "players hardly ever reach $70\%$ win rate" does not mean MtG has "less skill" (or smaller "skill separations / distances") than some other game where good players routinely reach e.g. $90\%$ win rate. It all depends on what constitutes a single "game". If a game is a single hand, the win rates would have a certain spread, but if a game is e.g. best-of-$7$-hands, the win rates would have a much wider spread and then maybe $70\%$ win rates would be much more common. Perhaps in MtG "game = 1 hand" is natural, but in many other card games generally agreed to be highly skill-based, e.g. poker and bridge, tournaments are not usually interested in "hand win percentage".