What's the significance of Mean Squared Error? Why not something else?

Question

Background: Masters in CS/Math.

I'm brushing up on statistics I see mean squared error everywhere. As a student I took it for granted, but now when I tried to find the reasons for why it's so prevalent I am told: simplicity, emphasis on outliers and mathematical properties like differentiability.

So what? It's not the only function with those properties. So why is it used so widely?

• Are there situations where it's provably the best function to use?
• Are there situations where there are other functions that are provably better to use?
• Say I am designing my own heuristic, and I have an error I want to minimize on. How do I know that squaring the error is the best way forward?

Answer 1

It has some other nice properties. For example, if you have a variable $X$ and you want to find the estimator $\hat{X}$ that minimizes: $$ \mathbb{E} [(X - \hat{X})^2] $$ the answer will be $\mathbb{E}[X]$, which is quite nice.

There are very useful results relating MSE and variance. For example, the minimum-variance unbiased estimator if it exists coincides with the MSE minimizer between the unbiased estimators. This allows us to reframe problems of minimizing variance into problems of minimizing MSE loss, which can be approached using optimization techniques.

Answer 2

The wide use of LMSE (Least Mean Square Error) is historical and conventional. The main advantage is to simplify the calculus in case of linear regression. This was a big advantage in the good old days when the numerical calulus were done by hand. This advantage is less important nowadays with the computers and available convenient softwares.

When linear regression with criterion LMSE is possible and well suited to a specific problem this is a real advantage compare to non-linear regression with other criteria (which involve iterative calculus starting from "guessed" values) not always well convergent nor robust.

LMSE is the holy grail in context of teaching but not always in practical applications. For example in case of data covering several different orders of magnitude. In this case LMSRE is generally more convenient.

Of course they are other criteria : For example Least Mean Square Relative Error, Least Mean Absolute Error or many others defined from an even function such as $f(0)=0$ : pp.4-5 in https://fr.scribd.com/doc/14819165/Regressions-coniques-quadriques-circulaire-spherique .

In practical applications there is no proof that LMSE is the most convenient. In each different case one should better define the criterion of fitting. This is rarely done. Loosely choosing LMSE is much easier !