Ground truth divided by prediction vs. mean average error for evaluation of a regression model.

103 Views Asked by At

I am familiar with the mean absolute error (MAE) for the evaluation of performances in a regression model. Given a set of ground truth (GT) values $y_i$, and a set of the model evaluated values $\hat{y}_i$, the mean average error $E_{mae}$ is defined by: $$E_{mae}=\frac{1}{N}\sum_{i=0}^N|y_i-\hat{y}_i|,$$ where $N$ is the number of points in the dataset, and the best error is $E_{mae}=0$. Recently, I have discussed with a colleague the following suggested error metric $E_{acc}$ (he called it accuracy): $$E_{acc} = \frac{1}{N}\sum_{i=0}^N\frac{y_i}{\hat{y}_i},$$ Where the best error $E_{acc}=1$. The claim here is that this metric is more informative since it also gives a sense of the direction of error. To be clearer, if this accuracy measure is averaged to a value greater than $1$, it means that the model is biased to evaluate a lower value than the GT.

I admit that I am unfamiliar with this type of metric. As far as I am aware, the accuracy measure is a measure for classification tasks and not regression. I could not find any such metric online.

Is there such a metric? Is it common? How does it compare with MSE? How does one encapsulate the direction of error in a regression model? Is there a third matric for that?

1

There are 1 best solutions below

1
On

Suppose $y_1 = y_2 = 1$ but you have predicted $\hat y_1 = -r$ and $\hat y_2 = +r$. Then $E_{acc} = 0$ but your model can actually be arbitrarily bad. A better way to identify bias in your predictions is plotting the fitted values against the residuals $r_i = y_i - \hat y_i$. Also note that if you include an intercept in regression, the residuals will always sum up to zero.