Questions Regarding Linear Regression

Question

Are the slope and intercept of a simple linear regression model always normally distributed? Is there ever a difference between the distribution of the estimated slope and intercept and the actual ones? I have only just begun learning about the subject but I am still not clear on the details. A final question: is the least squares method the same as linear regression in that it gives information like the $R^2$? Thanks!

Answer 1

• Are the slope and intercept of a simple linear regression model always normally distributed?

They are if the residuals are independent and normally distributed with common variance. More precise: The linear model is given by $\vec{y} = X\vec{\beta} + \vec{\varepsilon}$, where the design matrix $X$ is $n\times(k+1)$ with $\mathrm{rk}(X)=k+1 <n$ and $\vec{\beta}$ is a vector which contains the intercept and the slopes. Additionally we assume that $\vec\varepsilon \sim N(0, \sigma^2 I)$, that is, we assume that $\vec{y} \sim \mathrm{N}_n(X\vec{\beta}, \sigma^2 I)$. Then we have $\hat{\vec{\beta}}_{ML} \sim \mathrm{N}_{k+1}(\vec{\beta}, \sigma^2(X^T X)^{-1})$, i.e. the maximum likelihood estimator of the intercept and the slopes is multivariate normal.

• Is there ever a difference between the distribution of the estimated slope and intercept and the actual ones?

The actual ones are not random, so I don't understand your question here.

• Is the least squares method the same as linear regression?

The least squares method is a method that we apply to estimate the parameters, like the intercept and the slope. Quote from http://en.wikipedia.org/wiki/Linear_regression: Linear regression models are often fitted using the least squares approach, but they may also be fitted in other ways, such as by minimizing the "lack of fit" in some other norm (as with least absolute deviations regression), or by minimizing a penalized version of the least squares loss function as in ridge regression. Conversely, the least squares approach can be used to fit models that are not linear models. Thus, although the terms "least squares" and "linear model" are closely linked, they are not synonymous.