I have a data set that contains an actual value and two different forecasts for each datapoint. I want to compare the quality of each forecasting method.
I started by drawing a histogram of relative error for each method. e.g. method 1 histogram and method 2 histogram. It is clear visually that method 1 is better than method 2, but I want to be able to quantify this. I was hoping to use SD as one measure of quality, the problem is that the scale of the relative error is not symmetrical around zero. i.e. relative error can not be less than -1 but can extend infinitely in the positive direction. Using SD of this asymmetrical distribution makes forecasting methods that underestimate appear better than those that overestimate.
Experimenting I have developed the following formula for calculating an "errorfactor"
if forecast < actual
errorfactor = (-actual/forecast)+1
else
errorfactor = (forecast/actual)-1
This works on an intuitive level. A value of +0.5 means the forecast is 50% larger than the actual. A value of -0.5 means that the actual is 50% larger than the forecast.
I can plot a histogram of this distribution errorfactor histogram for method 2 and I can calculate SD for it.
My questions are
- Is there any statistical validity in what I have done?
- Is there a better way to assess variation in data of this sort?
p.s. I can see that when the errors are small it doesn't matter much, but it seems to in this case.
p.p.s. I have looked at MAPE and sMAPE but they don't really help.