Regression through different error types

Question

If I use either of these two forms of the error: $$R=\sum_{i=0}^{n-1} |{y_i-f(x_i)}|$$ or $$R=\sum_{i=0}^{n-1} ({y_i-f(x_i)})^2$$ to approximate the data, do i get the same solution to whatever form I choose for f(x) (the same in both cases)?

Answer 1

No you don't normally get the same solution (the same argmin) for different loss functions.

Minimizing sums of absolute residuals generally produces a different solution that minimizing sums of squared residuals.

For a simple example, consider the cases where $f$ is constant. Then for the loss based on squared residuals, your estimate of $f$ is the sample mean (this is usually shown using basic calculus, but you can do it with just a few lines of algebra, you don't need calculus), while for the loss based on absolute residuals your estimate of $f$ is any median (with even sample size, the usual sample median will lay at the center of any interval that minimizes the loss function).

You may like to check my claim with an example, such as $y_0=0, y_1=5, y_2=1$. For the squared residual loss, you get $R_2 = (f-0)^2+(f-5)^2+(f-1)^2 = 3f^2-12f+26$. This quadratic is minimized when $f=2$, which is indeed the sample mean. The loss based on absolute residuals (as a function of the value of $f$) is piecewise linear which has a minimum at $1$. We can see both loss functions as a function of the level of $f$ here (though $R_2$ has been shifted down by a constant in order that we can easily see both at the same time):