Imbalance due to ELO rating

Question

I want to use some rating system in my application. It will look like this:

$TotalRating = rating_1*significanceCoefficient_1 + rating_2*significanceCoefficient_2 + ... + rating_n*significanceCoefficient_n$

Significance coefficient - determines impact of a specific rating.

One of my ratings is ELO rating. All my other ratings(e.g. novelty of account, activity on the site) have the maximum value(e.g. 100), so, I can easily choose $significanceCoefficient$ for them. But ELO doesn't have a maximum, so I can't choose $significanceCoefficient$ for it.

This creates an imbalance - some users will be overrated due to very high ELO rating and other low ratings, while other users will have medium total rating due to medium ELO rating and high standard ratings.

I came up with a solution - I can set the maximum existing ELO rating of some user as 100% ELO percentage rating, and recalculate ELO percentage rating of other users based on their true ELO rating. E.g.:

User1 ELO = 1000 
User2 ELO = 100
User3 ELO = 200

Then:

User1 ELO percentage rating = 100%
User2 ELO percentage rating = 10%
User3 ELO percentage rating = 20%

It's an obvious solution, but not too elegant and requires permanent recount of the ELO percentage rating.

So, I'm looking for better solution for this problem. Do you have any ideas?

Answer 1

A simple approach is to set a maximum ELO that you will consider and round all higher ones down to that. Approaches that rescale $(0,\infty)$ to $(0,1)$ are not very different from this because large differences at the top end will be compressed to almost nothing.

One more general point to consider is that what counts is not the maximum for each component of the rating but the normal range. If one of your components has a maximum of $100$ but everybody has $99.5$ or higher it does not impact your total as much as one that has a maximum of $10$ but the scores range evenly from $0$ to $10$. Of course, you may want some components to be more important than others, but I would adjust the coefficients based on the range, not the maximum.

Answer 2

Perhaps build on what ILikeSerena suggests. Experiment with various functions $f$ that map $(0,\infty)$ to $(0,1)$. The function will have to have $1$ as a horizontal asymptote, which means the increases in a high ELO value won't change $f$ by much, but you can play with where the levelling off becomes most significant.

In fact decreasing the sensitivity of high scores to changes is built in to the ELO rating algorithms. You'd just be exaggerating that.

Check out relatives of the logistic function. You can translate, shift down and stretch so that they start at $f(0) = 0$. You can make the slope there whatever you like. It can be to the left or the right of the inflection point.

But ... if there are really wide variations in the ELO scores there's no obvious "fair" way to scale them. Perhaps assign them low relative weight among all the scores you are averaging.