I tried to find the best linear fit to $(1,-1)$, $(-1,1)$, $(-2,-2)$, and $(2,2)$. There is symmetry within the four points: across $y=x$ and $y=-x$. However, the least squares linear approximation for these four points gives a line $y=\frac{3}{5}x$. In this case, the normal equation has a unique solution. This is strange to me in two ways:
- Why is this line not symmetric across $y=x$?
- Why is $y=\frac{5}{3}x$ not a solution?
I guess this is some artifact of least squares that I have not fully understood. Any help with this matter is greatly appreciated.
This is because least squares minimizes the sum of squared vertical distances between the actual points and fitted points, i.e. it solves
$$\min_{\alpha,\beta} \sum_i (y_i-\alpha-\beta x_i)^2.$$
Minimizing the squared vertical distances is algebraically convenient and has many useful interpretations (including as a linear projection), which are a subject for another discussion.
Given this understanding, you can intuitively see why your suggested alternative of $y=\frac{5}{3}x $ would not work, as it would result in huge vertical gaps, particularly with the points $(1,-1)$ and $(-1,1)$ .