Average of a set of values using least squares formula

Question

To get the equation of a line $y = ax+b$ passing through a set of $n$ points $(x_i, y_i)$ using least squares formula, we have to solve the following system of linear equations to determine the coefficients, $a$ and $b$: $$\begin{cases} \sum_i{y_ix_i}-a\sum_i{x_i^2}-b\sum_i{x_i} = 0, \\ \sum_i{y_i}-a\sum_i{x_i}-bn= 0 \end{cases} $$ I want to use this formula to find the equation $y = b=\text{constant}$. That is, we substitute $a$ by $0$ in the system above to get: $$ \begin{cases} b = {\sum_i{y_ix_i}\over \sum_i{x_i}} \\ b = {\sum_i{y_i}\over n} \end{cases} $$

The second expression of $b$ is obvious.

Why did we get two expressions for $b$? which one should I use?

Answer 1

Least squares do not work like that. To obtain a least squares approximation (for instance linear regression) you must fix the form of the regression line right from the start. Setting $a=0$ in the formula for linear regression will not get you the formula for the regression with a constant function. This happens because the function to be minimized changes accordingly to the type of regression to be considered. Take for instance the data $(0,0), (1,2), (2,1)$. Least squares approximations by a constant function, a linear function and a quadratic function are given by

$$ y=1, \qquad y = \frac 12 + \frac 12 x, \qquad y = \frac 72 x - \frac 32 x^2, $$

respectively.

Answer 2

Your mistake is to consider that $a=0$ satisfies the equations. It does not.

You were looking for the minimum of the unexplained error, by canceling the gradient. If you arbitrarily set $a=0$, you are not getting the minimum.

---

The least-squares fit of the model $y=b$ is straightforward, leading to a single equation in a single unknown.