"Proof" of an Algebraic property of OLS Estimators

Question

I'm having a bit of trouble proving $\sum (x_i - \bar{x})\hat{e_i} = 0$. What I know so far is that the total sum of $\hat{e_i}$'s is zero by property of OLS so when you distribute the $\hat{e_i}$ in, one term "cancels out" and you are left with $\sum x_i\hat{e_i}$ which is equivalent to $\sum x_i(y_i-b_1-b_2x_i)$

When I attempt to simplify more, I keep getting stuck. What I'm doing so far is:

$\sum x_iy_i - b_1\sum x_i - b_2 \sum x_i^2$ = $\sum x_iy_i -(\bar{y}-b_2 \bar{x})\sum x_i - b_2 \sum x_i^2$

= $\sum x_iy_i - \bar{y}\sum x_i + b_2\bar{x}\sum x_i - b_2 \sum x_i^2$. And at this point I get stuck. Is there an easier trick? An OLS property I am forgetting? Thanks!

Answer 1

From your notation I assume that your true model is:

$$ Y_i=\beta_1+\beta_2 X_i + \epsilon_i \qquad i=1,\ldots,n $$ where $\beta_1$ and $\beta_2$ are the true parameters of the model. Then, if you assume that the (true) error term is iid and has zero conditional expectation you get (i.e., $E[\epsilon_i|X]=0$):

$$ E[Y_i|X_i]=\beta_1 + \beta_2 X_i + E[\epsilon_i|X_i]\\ =\beta_1+\beta_2 X_i $$ which is what you want to estimate using OLS. Let $b_1$ and $b_2$ be the OLS estimates of $\beta_1$ and $\beta_2$ respectively. Then, $$ b_2=\frac{\sum_{i=1}^{n} (Y_i-\bar{Y})(X_i-\bar{X})}{\sum_{i=1}^{n}(X_i-\bar{X})^2}\\ $$ Since, $$ \sum_{i=1}^{n} (Y_i-\bar{Y})(X_i-\bar{X})=\sum_{i=1}^{n} Y_i(X_i-\bar{X})-\bar{Y}\sum_{i=1}^{n}(X_i-\bar{X})\\ =\sum_{i=1}^{n} Y_i(X_i-\bar{X}) $$ and, $$ \sum_{i=1}^{n} (X_i-\bar{X})(X_i-\bar{X})=\sum_{i=1}^{n} X_i(X_i-\bar{X})-\bar{X}\sum_{i=1}^{n}(X_i-\bar{X})\\ =\sum_{i=1}^{n} X_i(X_i-\bar{X}) $$ because $\sum_i(X_i-\bar{X})=0$ you can rewrite $b_2$ as follows: $$ b_2=\frac{\sum_{i=1}^{n} Y_i(X_i-\bar{X})}{\sum_{i=1}^{n}X_i(X_i-\bar{X})}\\ $$ Let $e_i$ be the OLS residual, i.e., $e_i=Y_i-b_1-b_2X_i$, $i=1,\ldots,n$. Then, $$ \sum_i(X_i-\bar{X})e_i=\sum_i(X_i-\bar{X})(Y_i-b_1-b_2X_i)\\ =\sum_i Y_i(X_i-\bar{X})-b_1\sum_i(X_i-\bar{X})-b_2\sum_i X_i(X_i-\bar{X})\\ =\sum_i Y_i(X_i-\bar{X})-b_2\sum_i X_i(X_i-\bar{X})\\ =\sum_i Y_i(X_i-\bar{X})-\left(\frac{\sum_i Y_i(X_i-\bar{X})}{\sum_i X_i(X_i-\bar{X})}\right)\sum_i X_i(X_i-\bar{X}) $$ where again I have used the fact that $\sum_i (X_i-\bar{X})=0$.

Answer 2

Assume you have a model of the following form: $y=X\beta+\epsilon$ where $X$ is a $n$ by $K$ matrix of $K$ explanatory variables and $n$ observations of each variable. $y$ a column vector consisting of the $n$ observations of the dependent variable and $\epsilon$ is a column vector consisting of the error terms. In your case it seems like you have a model where $X=[1,X_1]$ (i.e. one explanatory variable and an intercept). Now, I assume you know how to find the normal equation when solving for the least squares estimate $b$ of $\beta$. The normal equation reads: $$X^te=0$$ where $e=y-Xb$. Now by simply multiplying out the normal equation above your claim follows, since $0=X^te=[\sum{e_i},\sum{x_ie_i}]^t$. Hence $\sum{(x_i-\bar{x})e_i}=\sum{x_ie_i}-\bar{x}\sum{e_i}=0$.

Also note that the result above applies to any multivariate linear regression model as long as you have an intercept term. However without the intercept term (also called a regression through the origin) we can not be sure that $\sum{e_i}=0$ and hence your claim may no longer be true.