I have a question about weights, and maybe the best way to ask is to explain the background of what I'm trying to do.
There is software that scans computer file systems for certain faults including things like security vulnerabilities. Each detected fault is assigned a severity ranking on a 5-point scale (5 being highest priority) by the software. Thousands of faults are typically detected, and each needs to be reviewed and eventually fixed. However, we can't rely on only the scanner severity rankings to determine fix-priority, because other attributes affect the overall seriousness of a fault. For example, some attributes are assessed with different scanning software, and some are manually assessed by IT teams, etc. Let's say there are four external attributes: attr1, attr2, attr3, attr4, and each has its own severity ranking system. attr1 and attr2 use a 5-point scale with 1/very low, 2/low, 3/med, 4/high, 5/urgent. attr3 uses a 3-point scale of low/med/high, and attr4 is a simple yes or no. I am trying to create a metric for fix-priority based on a 10-point scale, with 10 being the most urgent, based on the scanner ranking and taking into account these external attributes. My thinking is that, for a fault to receive the most urgent fix-priority=10, the scanner ranking would need to be at its max value (5) and each external attr would also need to be assessed at its max value. Any ranking less than maximum would cause the calculated fix-priority to decrease to a number lower than 10.
My formula is this:
$( ( (scannerRank*0.2)+(attr1*0.2)+(attr2*0.2)+(attr3*0.33)+(attr4*0.5) ) / 4.99 ) * 10$
It seems to work. But I have two questions:
attr4 uses a simple yes or no metric, so each of its values, 1 or 2, is worth 0.5 in my calculation. Similarly, attr3 has 3 possible values, each worth 0.33. So, are attr3 and attr4 getting more weight and importance in the calculation compared to attr1,attr2 and scannerRank? If so, how should I handle this, if I viewed each attr as being equally important?
In the future, I may change my mind about each attr having equal importance - I may want to view certain ones as being more or less severe than another. How would the formula need to change to accomplish that?
For simplicity, I would convert each ranking to a number between $0$ and $1$ by dividing by the length of the scale. For the yes/no attribute the values would be either $0$ or $1$. Call the five severities $s_i$. Let weights $p_i$ sum to $1$, and calculate $$ S = p_1s_1 + p_2s_2 + p_3s_3 + p_4s_4 + p_5s_5 . $$ You'll get $S=1$ only when things are as severe as possible in all categories.
Weighting each attribute equally is just setting every $p_i = 0.2$. You can play with the weights to get a balance that captures your sense of the relative importance of the attributes.