intuition behind having a unique regression line

187 Views Asked by At

I understand this mathematically. we have function of 2 variables represents the sum of square errors. We have to find the $a$ and $b$ that minimize the function. there is only one minimum point.

But when I think of it, I can't see why 2 different lines would not bring the value to the same minimum. we have 2 degrees of freedom, so why can't we find a new pair of $(a,b)$ that will have the same value.

2

There are 2 best solutions below

0
On BEST ANSWER

You have two degrees of freedom (the slope $m$ and $y$-intercept $b$ of the regression line, say), but also two constraints: The partial derivatives of the squared total error $E$ with respect to $m$ and with respect to $b$ must vanish. That means you expect only finitely many (local) minima. (As A.E. says, $E$ is strictly convex, so there is at most one minimum.)

0
On

Basically, as you said, you are minimizing sum of squares, i.e., $$ \underset\beta{\arg\min} \sum_{i=1}^n(y_i-\beta_0 - \sum_{j=1}^p\beta_jx_j)^2 = \underset\beta{\arg\min}||y-X\beta|| $$ which has to be strictly convex function over the parametric space $\mathcal{B}$, otherwise the Hessian matrix of this quadratic form, $X'X$, won't be positive definite, so you won't have its inverse $(X'X)^{-1}$ and thus (you won't have) the solving vector of the form $$ \beta = (X'X)^{-1}X'y . $$ Sure you have pseudo-inverse matrix that can give you a $\beta$ where there is no unique solution, but in the straightforward basic case, if you have a solution - it will be unique.