derivation of simple linear regression parameters

Question

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial derivatives (although I might want to do it wituout partials if it's easier).

I've posted my attempt below. I don't know what to from here.

Answer 1

The partial derivatives of $$ Q(\alpha,\beta)=\sum_i (y_i-\alpha-\beta x_i)^2 $$ are $$ \frac{\partial Q}{\partial \alpha}(\alpha,\beta)=-2\sum_i(y_i-\alpha-\beta x_i)=-2(n\bar{y}-n\alpha-n\beta\bar{x})\tag{1} $$ and $$ \frac{\partial Q}{\partial \beta}(\alpha,\beta)=-2\sum_ix_i(y_i-\alpha-\beta x_i)=-2(\mathrm{SP}_{xy}-n\alpha\bar{x}-\beta\mathrm{SS}_x)\tag{2} $$ with $\mathrm{SP}_{xy}=\sum x_i y_i$ and $\mathrm{SS}_x=\sum x_i^2$. Putting $(1)$ equal to zero gives us $$ \hat{\alpha}=\bar{y}-\hat{\beta}\bar{x} $$ and plugging this into $(2)$ gives us the equation $$ \begin{align} 0=\mathrm{SP}_{xy}-n(\bar{y}-\hat{\beta}\bar{x})\bar{x}-\hat{\beta}\mathrm{SS}_x&=\mathrm{SP}_{xy}-n\bar{y}\bar{x}+n\hat{\beta}\bar{x}^2-\hat{\beta}\mathrm{SS}_x\\ &=\mathrm{SP}_{xy}-n\bar{y}\bar{x}-\hat{\beta}(\mathrm{SS}_x-n\bar{x}^2) \end{align} $$ and hence $$ \hat{\beta}=\frac{\mathrm{SP}_{xy}-n\bar{y}\bar{x}}{\mathrm{SS}_x-n\bar{x}^2}. $$

Answer 2

The principle underlying least squares regression is that the sum of the squares of the errors is minimized. We can use calculus to find equations for the parameters $\beta_0$ and $\beta_1$ that minimize the sum of the squared errors, $S$.

$$S = \displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum \left(y_i - \hat{y_i} \right)^2= \sum \left(y_i - \beta_0 - \beta_1x_i\right)^2$$

We want to find $\beta_0$ and $\beta_1$ that minimize the sum, $S$. We start by taking the partial derivative of $S$ with respect to $\beta_0$ and setting it to zero.

$$\frac{\partial{S}}{\partial{\beta_0}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-1) = 0$$ $$\sum \left(y_i - \beta_0 - \beta_1x_i\right) = 0 $$ $$\sum \beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$n\beta_0 = \sum y_i -\beta_1 \sum x_i $$ $$\beta_0 = \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i \tag{1}$$ $$\beta_0 = \bar y - \beta_1 \bar x \tag{*} $$

now take the partial of $S$ with respect to $\beta_1$ and set it to zero.
$$\frac{\partial{S}}{\partial{\beta_1}} = \sum 2\left(y_i - \beta_0 - \beta_1x_i\right)^1 (-x_i) = 0$$ $$\sum x_i \left(y_i - \beta_0 - \beta_1x_i\right) = 0$$ $$\sum x_iy_i - \beta_0 \sum x_i - \beta_1 \sum x_i^2 = 0 \tag{2}$$ substitute $(1)$ into $(2)$ $$\sum x_iy_i - \left( \frac{1}{n}\sum y_i -\beta_1 \frac{1}{n}\sum x_i\right) \sum x_i - \beta_1 \sum x_i^2 = 0 $$ $$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i + \beta_1 \frac{1}{n} \left( \sum x_i \right) ^2 - \beta_1 \sum x_i^2 = 0 $$

$$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i = - \beta_1 \frac{1}{n} \left( \sum x_i \right) ^2 + \beta_1 \sum x_i^2 $$ $$\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i = \beta_1 \left(\sum x_i^2 - \frac{1}{n} \left( \sum x_i \right) ^2 \right) $$ $$\beta_1 = \frac{\sum x_iy_i - \frac{1}{n} \sum x_i \sum y_i}{\sum x_i^2 - \frac{1}{n} \left( \sum x_i \right) ^2 } = \frac{cov(x,y)}{var(x)}\tag{*}$$