Covariance of least squares parameter?

Question

How do we derive the $\mathbb{E}(\hat{\beta}_{LS})$ and $\textbf{Cov}(\hat{\beta}_{LS})$. How does the randomness of ${\epsilon_i}$ contribute to this?

Answer 1

Suppose the model is $Y=X\beta+\epsilon$, where $Y=(Y_1,...,Y_n)'$ are the responses from $n$ measurements, and $X$ is the $n\times p$ matrix of corresponding explanatory variables. The least square estimate is in the following form $$\hat\beta_{LS}=(X'X)^{-1}X'Y. $$

The randomness of $Y$ comes from $\epsilon$. If we assume $\mathbb E(\epsilon) = 0$ and $\text{cov}(\epsilon)=\Sigma$, You can see that $$\mathbb E \hat\beta_{LS} = (X'X)^{-1}X'(X\beta)=\beta;$$ $$\text{cov}(\hat\beta_{LS}) = (X'X)^{-1}X'\Sigma X(X'X)^{-1}$$. We used the fact that for any matrix $A$ and random vector $X$, $$\mathbb{E}(AX)=A\mathbb{E}(X);$$ $$\text{cov}(AX)=A\text{cov}(X)A'. $$