Find $\operatorname{Cov}(\hat{\beta}_0, \hat{\beta}_1)$.

7.4k Views Asked by At

Let $Y_1,Y_2,\ldots,Y_n$ and $X_1,X_2,\ldots,X_m$ be random variables with $E(Y_i)=\mu_i$ and $E(X_j)=\xi_j$. Define $$U_1=\sum_{i=1}^n a_i Y_i\quad\text{and}\quad U_2=\sum_{j=1}^m b_j X_j$$ for constants $a_1,a_2,\ldots,a_n$ and $b_1,b_2,\ldots,b_m$. Then the following hold:

$\quad\textbf a\,\,$ $E(U_1)=\sum_{i=1}^n a_i\mu_i.$

$\quad\textbf b\,\,$ $V(U_1)=\sum_{i=1}^n a_i^2 V(Y_i)+2\sum\sum_{1\leqslant i \leqslant j \leqslant n} a_i a_j \operatorname{Cov}(Y_i,Y_j)$, where the double sum is over all pairs $(i,j)$ with $i\lt j$.

$\quad\textbf c\,\,$ $\operatorname{Cov}(U_1, U_2)=\sum_{i=1}^n \sum_{j=1}^m a_i b_j \operatorname{Cov}(Y_i, X_j)$.

I am also given a hint to use this which I do not know how I can apply this to this question? Suppose that $Y_1, Y_2, \ldots, Y_n$ are independent normal random variables with $E(Y_i) = \beta_0 + \beta_1 x_i$ and $V(Y_i) = \sigma^2$, for $i = 1, 2, \ldots, n$. The maximum-likelihood estimators of $\beta_0$ and $\beta_1$ are the same as the least-squares estimators of enter image description hereFind $\operatorname{Cov}(\hat{\beta}_0, \hat{\beta}_1)$.

What I got so far is $$\operatorname{Cov}(\hat{\beta_0}, \hat{\beta}_1) = \operatorname{Cov}(\bar{Y}-\hat{\beta}_1\bar{x}, \hat{\beta}_1)$$

How can I move forward with this?

2

There are 2 best solutions below

12
On BEST ANSWER

This assumes that the $x_{i}$ are fixed, the model is quite different if they are not. However, this assumption is reasonable given the expected value and variance of the $Y_{i}$. Usually, for any linear relationship, we have the model $$Y_i=\beta_0+\beta_1 x_i+\epsilon_i,$$ where $\beta_{0},\beta_1 \in \mathbb{R}$, the $x_i$ are fixed, and $\mathbb{E}(\epsilon_i)=0, \text{Var}(\epsilon_i)=\sigma^2$ for a constant $\sigma^{2}$. This is very similar to your context, so I will assume this is the correct context. Denote the random variables $Y_i$ with lower case $y_i$ for consistency with typical regression notation. We have $$\text{Cov}(\hat{\beta}_0,\hat{\beta}_1)=\text{Cov}(\bar{y}-\hat{\beta_{1}}\bar{x},\hat{\beta}_1) \\=\text{Cov}(\bar{y},\hat{\beta}_1)-\text{Cov}(\hat{\beta}_1\bar{x},\hat{\beta}_1)\\=\text{Cov}(\bar{y},\hat{\beta}_1)-\bar{x}\text{Cov}(\hat{\beta}_1,\hat{\beta}_1)\\=\text{Cov}(\bar{y},\hat{\beta}_1)-\bar{x}\text{Var}(\hat{\beta}_1),$$ where the third equality comes from the fact that the $x_i$ are fixed and the fourth equality comes from the definitions of variance and covariance. From here, substitute the given expression for $\hat{\beta}_1$ (this expression and the one given for $\hat{\beta}_0$ can be derived from the usual method of maximum likelihood or from the least-squares estimators, they are equivalent), and use your knowledge about covariances and variances to finish the calculation (some information that you need will come from the proposition - "the following" in your box). Important hint: you will need to write $\hat{\beta}_1$ as a linear combination of the $y_{i}$, this is the key to completing the computation. Note that the covariance function and the vector space of random variables form an inner product space, so $\text{Cov}(\cdot,\cdot)$ is linear in the first argument (and second argument as a result of the symmetry clause of inner products) and thus, $$\text{Cov}\left(\sum_{i=1}^n c_i Z_i,X\right)=\sum_{i=1}^n c_i \text{Cov}(Z_i,X)$$ (this will be very useful as well) for constants $c_1,c_2,\ldots,c_n$ and any random variables $Z_1,Z_2,\ldots,Z_n,X$. Also, here is yet another two useful identities/derivations:$$\sum_{i} x_i(x_i - \bar{x})=\sum_i x_i(x_i - \bar{x}) - \bar{x} \sum_i(x_i - \bar{x})\\=\sum_i(x_i - \bar{x})(x_i-\bar{x})=\sum_i (x_i - \bar{x})^2= S_{xx},$$ and similarly, $$\sum_i x_i(y_i - \bar{y})=\sum_i x_i(y_i - \bar{y}) - \bar{x} \sum_i(y_i - \bar{y})\\=\sum_{i}(x_i - \bar{x})(y_i-\bar{y})= S_{xy},$$ since we know that $\sum_i(x_i-\bar{x})=\sum_i(y_i-\bar{y})=0$.

0
On

$cov(\hat{\beta_0}, \hat{\beta_1})=cov (\hat{y}-\hat{\beta_1}\bar{x}, \hat{\beta_1})$

$=E[(\bar{y}-\hat{\beta_1}\bar{x})(\hat{\beta_1})]-E[\hat{y}-\hat{\beta_1} \bar{x}].E[\hat{\beta_1}]$

$=E[\hat{\beta_1}\bar{y}-\bar{x}\hat{\beta_1^2}]-(\hat{y}-\bar{x}\beta_1) (\beta_1)$

$=\bar{y}E[\hat{\beta_1}]-\bar{x}E[\hat{\beta_1^2}]-\bar{y}\beta_1+\bar{x}E[\hat{\beta_1}].E[\hat{\beta_1}]$

$=\bar{y}\beta_1-\bar{x}E[\hat{\beta_1^2}]-\bar{y}\beta_1+\bar{x}E[\hat{\beta_1}].E[\hat{\beta_1}]$

$=-\bar{x}(E[\hat{\beta_1^2}]-E[\hat{\beta_1}].E[\hat{\beta_1}])$

$=-\bar{x}. Var[\hat{\beta_1}]$

Variance of $\beta_1$ can be easily found in textbooks or online help. So I am omitting it.

$=-\bar{x}[\frac{\sigma ^2}{S_{xx}}]$