Prove that explanatory variable $x_i$ is uncorrelated with residual $e_i$

Question

In simple linear model, if we want to prove that $x_i$ is uncorrelated with the residual $e_i$, we will do that by showing that $\sum_{i=1}^n X_i e_i=0.$ But why that implies explanatory variable and residual are uncorrelated? Shouldn't the sample covariance between $x$ and $e$ be $\sum_{i=1}^n \frac{(X_i-\bar{X})(e_i-\bar{e})}{n-1}=0$? So how to understand that $\sum_{i=1}^n X_ie_i=0$ implies covariance between $X$ and $e$ be 0?

Answer 1

You can simply write out the sample covariance: $$\begin{align}&\sum_{i=1}^{n}{\frac{(X_i-\bar{X})(e_i-\bar{e})}{n-1}}=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar{X})(e_i)\\&=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar{X})(e_i)\tag{$\bar{e}=0$ }\\&=\frac{1}{n-1}\left(\sum_{i=1}^{n}X_ie_i+\sum_{i=1}^{n}\bar{X}e_i\right)\\&=\frac{1}{n-1}\left(\sum_{i=1}^{n}X_ie_i+\bar{X}\sum_{i=1}^{n}e_i\right)\tag{ $\bar{X}$ is constant}\\&=\frac{1}{n-1}\left(\sum_{i=1}^{n}X_ie_i+\bar{X}\bar{e}\right)\tag{definition of $\bar{e}$}\\&=\frac{1}{n-1}\sum_{i=1}^nX_ie_i.\tag{$\bar{e}=0$}\end{align}$$

So the covariance being equal to zero is equivalent to $\sum_{i=1}^{n}X_ie_i=0.$

Answer 2

First let us be clear that there is a difference between the residual $e_i$ and the error $\varepsilon_i.$

The errors are typically taken to be independent and identically distributed; there are the amounts by which the observed $y$-values differ from their expected values.

The residuals, on the other hand are the amounts by which the observed $y$-values differ from the least-squares estimates of their expected values. This implies the sum of the residuals must be $0.$ (It also implies the residuals are correlated with each other in a way that depends on the $x$-values. But that part need not concern us here except as another point underscoring the fact that they differ from the errors, which are not correlated with each other.)

Thus we have $\overline e=0.$ And therefore $$ \sum_{i=1}^n (x_i-\overline x)(e_i -\overline e) = \sum_{i=1}^n (x_i-\overline x) e_i = \left( \sum_{i=1}^n x_i e_i\right) - \overline x \sum_{i=1}^n e_i = \left( \sum_{i=1}^n x_i e_i\right) - 0. $$ The reason $\overline x$ can be pulled out of one of the sums is that it does not change as $i$ goes from $1$ to $n.$