There is a problem in the MIT OCW 18.02 (Multiple Variable Calculus) course that could be stated in the following way.
Show that the equations:
$$ \left\{ \begin{array}{c} (\sum {x_i^2})a + (\sum x_i) b=\sum {x_i y_i} \\ (\sum x_i)a + nb=\sum y_i \end{array} \right. $$
for the method of least squares have a unique solution, unless all the $x_i$ are equal. Explain geometrically why this exception occurs.
Hint: use the fact that for all values of $u$, we have $\sum_1^n {(x_i - u)^2} \ge 0$, since squares are always non-negative. Write the left side as a quadratic polynomial in $u$. Usually it has no roots. What does this imply about the coefficients? When does it have a root? (Answer these two questions by using the quadratic formula.)
The course assumes only basic exposition to the linear algebra, so I expect it to be solved in a simple algebraical approach without using matrix expressions (that's how Linear Least Squares are explained in the course).
I tried two approaches and in both I get stuck:
- In order for the system to have a unique solution the determinant of the coefficient matrix should be non-zero, thus:
$$n(\sum x_i^2) - (\sum x_i)^2 \ne 0$$
I see that it this condition fails when $x_i = x$ for all $i$ (so to say, all $x_i$ are equal). However I cannot prove the converse.
- Starting from the hint, if I rewrite $\sum_1^n (x_i - u)^2$ as a quadratic polynomial in $u$, I get $$\sum_1^n (x_i - u)^2 \\ = \sum_1^n (x_i^2 - 2 x_i u + u^2) \\ = \sum_1^n x_i^2 - (2 \sum_1^n x_i) u + n u^2 $$
So the discriminant is:
$$D = 4 (\sum_1^n x_i)^2 - 4 n \sum_1^n x_i^2 \\ = 4\left( (\sum_1^n x_i)^2 - n \sum_1^n x_i^2 \right) $$
and in order for the roots in $u$ to exist, it must be $D \ge 0$.
With this method I cannot understand both how it is connected with the equations and how the discriminant inequality can be resolved.
Source: see Supplemental Problems 2G-2 at the course page.