Investigate whether $\widetilde\beta$ is an unbiased estimator for $\beta$

49 Views Asked by At

Consider a linear model $Y=X\beta+\varepsilon$ , where $Y,\varepsilon \in \Bbb R^n,\beta\in \Bbb R^p$ and with model matrix $X \in \Bbb R^{n×p}$ of full rank, $n, p\in \Bbb N$ with $1\lt p\le n$.Consider further the singular value decomposition of the model matrix X,$$X=UDV^T$$, where U is $n×p$, D is a $p×p$ diagonal matrix and V is $p×p$.(Note: The diagonal elements of $D=diag(\lambda_1,\lambda_2,\ldots,\lambda_p)$ are the positive square roots of the eigenvalues of $X^TX$ or $XX^T$ and U and V contain normalized eigenvectors of $XX^T$ and $X^TX$ respectively, $i.e.U^TU=I,V^TV=I.$ We assume $\lambda_1\ge\lambda_2\ge\ldots\ge\lambda_p.)$

  1. Show that the least square estimator $\hat \beta$ for $\beta$ can be written as $\hat \beta=V D^{−1}U^TY$. An alternative estimator for $\beta$ is given by $\widetilde\beta:=V D^{−1}_* U^TY$, where $D^{−1}_*=diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)$ for some $1\le k \lt p$. (Note: Such an estimator can e.g. be useful if some covariates are highly correlated and $X^TX$ is close to singular.)
  2. Investigate whether $\widetilde\beta$ is an unbiased estimator for $\beta$. If applicable, calculate its bias.

Answer:2)To investigate whether the alternative estimator $\beta$ equals $VD^{-1}U^TY$ is unbiased, we need to check whether its expected value equals the true parameter vector $ẞ$.Taking the expectation of $\beta$ , we have $\Bbb E (\widetilde{\beta}) = \Bbb E(VD^{-1}_* U^TY)$.

Since $D^{-1}_* $ is a diagonal matrix with non-zero elements only on the first k diagonal positions, we can write $D=diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)$ Therefore, $\widetilde\beta =(VD^{-1}_* U^TY)$ can be written as $\widetilde\beta = Vdiag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)U^TY)$. Expanding this expression, we have $\widetilde \beta = (V_1, V_2,\ldots,V_p)diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)U^TY)$,where $V_1, V_2,\ldots,V_p$ represent the columns of V. Taking the expectation, we have, $\Bbb E (\widetilde{\beta})=\Bbb E (V_1, V_2,\ldots,V_p)diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)U^TY)$. Since the expectation operator is linear, we can move it inside the product:$\Bbb E (\widetilde{\beta})= (V_1, V_2,\ldots,V_p) \Bbb Ediag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)U^TY)$.Since the diagonal elements of $D^{-1}_*$ only have non-zero values on the first k positions,$\Bbb E D^{-1}_* = diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)$ Therefore,$\Bbb E (\widetilde{\beta})= (V_1, V_2,\ldots,V_p) diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)\Bbb E U^TY)$. By the properties of the expectation operator,$\Bbb E U^TY=U^T\Bbb EY$. But $\Bbb E (Y) = X ẞ$, so $\Bbb E U^TY = U^T X ẞ$.Therefore,$\Bbb E (\widetilde{\beta})= (V_1, V_2,\ldots,V_p) diag(\lambda^{-1}_1,\lambda^{-1}_2,\ldots,\lambda^{-1}_k,0,\ldots,0)\Bbb E U^TX ẞ)$. Since the matrix product diag equals zero for any matrix X, except for the first k positions, we have,$\Bbb E (\widetilde{\beta})= (V_1, V_2,\ldots,V_p) 0 $. In other words, the alternative estimator $\widetilde{\beta}$ is unbiased and its expected value is equal to the true parameter vector ẞ.

i just try it but i am not sure so need help for the solution verification.