What's in a name? (Sum of Squares)

Question

I have always believed that in order to fully understand and appreciate the mathematical subtleties and ideas behind a certain concept is to understand the name given to it.

For now, can someone please help me understand what 'RESIDUAL SUM OF SQUARES' means? How is this different from OTHER sum of squares? Like 'LEAST sum of squares', etc?

What is meant by 'RESIDUAL'?

Thanks

Answer 1

Residual refers to the difference between the $observed$ and $predicted$ value. Given some model function $f$, that $predicts$ value $f(x_i)$ while the one actually observed is $y_i$, thus there is a $residual$ given by $y_i-f(x_i)$ and $RSS$ or residual sum of squares is given by

$$ RSS = \sum_i(y_i-f(x_i))^2 $$ RSS is indicative of how overall fit of model(lesser is better)

For a simple linear regression where $f(x)$ predicts the observations using from a linear model $\beta_0 + \beta_1x$. The coefficients $\beta_0, \beta_1$ are thmeselves determined using Ordinary least square estimation wherein you choose most optimal values of $\beta_0, \beta_1$ such that your RSS is minimized, resulting in a good model fit.