Population OLS coefficient in simple regression?

Question

The population OLS coefficient for some $X_i \in \mathbb{R}^d, Y \in \mathbb{R}$ for the model $Y = \beta’X + e$ is defined as $$ \beta =\mathbb{E}[X_iX_i']^{-1}\mathbb{E}[X_iY_i] $$ and if $X$ is a scalar random variable, the commonly shown formula for simple regression is $$ \beta = Var(X_i)^{-1}Cov(X_i,Y_i) $$ But from the more general first equation, shouldn’t this be $$ \beta = \mathbb{E}[X_i^2]^{-1} E[X_i Y_i] $$ instead? Where is the variance / covariance coming from? (I.e. how do you derive that popular variance covariance formula from the more general vectorized version)? I am clearly missing something, but I can’t seem to see what is happening to the extra terms (i.e. since $Var(X) = E[X^2] + E[X]^2$, not just $E[X^2]$). Is there something conceptually obvious that I am overlooking?

Answer 1

I think you cannot get the results from a simple regression model because you did not include the intercept term in your model.
The common formula for a simple regression model is derived from the model $Y = \beta_0 + \beta_1X + \varepsilon$. This can be rewritten in matrix form as $Y = \beta^\top U + \varepsilon$, where $\beta = (\beta_0, \beta_1)^\top$ and $U = (1_n, X)$. For simplicity, let's assume we observe $n$ tuples of $(x_i,y_i)$. The OLS estimation of $\beta$ yields $\hat{\beta}=(U^\top U)^{-1}U^\top Y$ and \begin{align*} \begin{pmatrix} \hat{\beta_0} \\ \hat{\beta_1}\end{pmatrix} &= \begin{pmatrix} n & \sum_{i=1}^n x_i \\ \sum_{i=1}^nx_i & \sum_{i=1}^nx_i^2\end{pmatrix}^{-1} \begin{pmatrix}\sum_{i=1}^ny_i \\ \sum_{i=1}^nx_iy_i\end{pmatrix} \\ & = \dfrac{1}{n(\sum_{i=1}^n(x_i-\bar{x})^2}\begin{pmatrix} \sum_{i=1}^nx_i^2 & -\sum_{i=1}^n x_i \\ -\sum_{i=1}^nx_i & n\end{pmatrix} \begin{pmatrix}\sum_{i=1}^ny_i \\ \sum_{i=1}^nx_iy_i\end{pmatrix}\\ & = \begin{pmatrix}\bar{y}-\hat{\beta_1}\bar{x} \\ \dfrac{\sum_{i=1}^n (x_i-\bar{x})(y_i - \bar{y})}{\sum_{i=1}^n(x_i - \bar{x})^2}\end{pmatrix} \\ & = \begin{pmatrix}\bar{y}-\hat{\beta_1}\bar{x} \\ \dfrac{Cov(X,Y)}{Var(X)}\end{pmatrix} \ , \end{align*} where $Cov(X,Y), Var(X)$ is the sample covariance and sample variance respectively. We can also see that it holds true in a population sense (assumed both $X, Y$ are random variables): \begin{align*} Cov(X,Y) & = Cov(X,\beta_0 + \beta_1 X + \varepsilon) \\ & = 0 + \beta_1Var(X) + 0 \\ \implies\beta_1 &= \dfrac{Cov(X,Y)}{Var(X)}\end{align*}

In your case, you assume that $X$ is a scalar, which means that your model is $Y = \beta_1 X +\varepsilon$, which is different from the classic simple regression model as the intercept term is missing. In this case, the OLS estimation is \begin{align*}\hat{\beta_1} &= (X^\top X)^{-1} X^\top Y \\ & = \dfrac{\sum_{i=1}^n x_iy_i}{\sum_{i=1}^n x_i^2} \end{align*} In a population sense, we still have \begin{align*} Cov(X,Y) & = Cov(X,\beta_1 X + \varepsilon) \\ & = \beta_1 Var(X) \\ \implies\beta_1 &= \dfrac{Cov(X,Y)}{Var(X)}\end{align*}