Reduce a range of values

1.1k Views Asked by At

I am using this weighted sum from the question How to build a function that gives recent years higher weight?

To reduce the value I tried square root of $X,$ then tried the $8$th root, $\sqrt{\sqrt{\sqrt{X}}}$ which is $X^{\frac{1}{8}}$ and get better results.

My question is how can I justify this use? Can I know why I got better results when using the $8$th root or is it just random, depend on data only? And how can I know if there is a better way of doing it?

1

There are 1 best solutions below

2
On BEST ANSWER

There is extensive literature on data transformations, specifically to lower the variance of the residuals contained in the data, fit to a given model.

I'm not certain if it is exactly what you need, but the square root transformation is a well-known variance stabilizer.

From a cursory review of your data, I think you might be able to look into various types of regression analysis, and that would lead you into all sorts of ways to model your data, all of which have rich and rigorous justifications for such transformations.

Even more simply, you could use an exponentially-decaying rate to arrive at your weights.

The point to remember is that with your current methodology, you are just shrinking the values by multiplying them by smaller and smaller numbers. If you wanted a really small number you could use $x^{1/100}$ but you are correctly deducing that there isn't much of a justifiable reason (mathematically) to do so.

Keep experimenting with things...for one, I would try weighting each score by a factor of $x^{\frac1t}$ where t is the number of years ago. For example

Year 1: 10 movies, 5 comedies Year 2: 8 movies, 2 comedies Year 3: 10 movies, 7 comedies

take your sample proportions (Year 1= .5, Year 2=.25 and Year 3=.7) and apply the following weights:

Year 1= $(.5)^{\frac13}$

Year 2= $(.25)^{\frac12}$

Year 3= $(.7)^{\frac11}$

This would down-weight past years in a more "tidy" way.