Show that $\hat{\beta} - \hat{\beta}_{(i)} = \frac{(X^TX)^{-1}\vec{x}_{i}^{T}(y_i-\hat{y}_i)}{1-h_{ii}}$

Question

$$\hat{\beta} - \hat{\beta}_{(i)} = \frac{(X^TX)^{-1}\vec{x}_{i}^{T}(y_i-\hat{y}_i)}{1-h_{ii}}$$

where $h_{ii} = \vec{x}_{i}^T (X^TX)^{-1} \vec{x}_{i}^{T}$ and $\hat{\beta}_i$ is the estimator of $\hat{\beta}$ with the i-th observation excluded. Note that we are using the linear model $y = X\beta +\epsilon$.

I tried rewriting the betas, but didn't get anywhere. Any guidance is appreciated.

Answer 1

Let $x_k^T$ denote the k^th row of $X$.

So $$ X = \begin{pmatrix} x_1^T \\ x_2^T \\ \vdots \\ x_n^T \end{pmatrix}, $$

We have $$X^TX = \sum_{i=1}^n x_i x_i^T.$$

Let $X_{(i)}$ denote the matrix obtained from deleting the i^th row of $X$.

Then clearly, $$ X_{(i)}^TX_{(i)} = X^T X - x_ix_i^T,$$

inverting and using Sherman-Woodbury-Morrison formula on the RHS we get $$ \label{e:1} (X_{(i)}^TX_{(i)})^{-1} = (X^T X)^{-1} + \frac{ (X^T X)^{-1} x_i x_i^T (X^T X)^{-1} } {1 - x_i^T (X^TX)^{-1} x_i}. \tag{*} $$

Let $$Y = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{pmatrix}$$ denote the observation vector and let $Y_{(i)}$ denote the column vector obtained after deleting the the i^th row of $Y$.

Now, $$ \label{e:2} X_{(i)}^T Y_{(i)} = X^T Y - x_i y_i \tag{#}. $$

Since $$ \hat{\beta}_{(i)} = (X_{(i)}^TX_{(i)})^{-1} X_{(i)}^T Y_{(i)}$$ we have from $\eqref{e:1}$ and $\eqref{e:2}$ that $$ \hat{\beta}_{(i)} = \left( (X^T X)^{-1} + \frac{ (X^T X)^{-1} x_i x_i^T (X^T X)^{-1} } {1 - x_i^T (X^TX)^{-1} x_i} \right) \left( X^T Y - x_i y_i \right)$$

or $$ \begin{align} \hat{\beta}_{(i)} &= (X^T X)^{-1}\left( X^T Y - x_i y_i \right) + \frac{ (X^T X)^{-1} x_i x_i^T (X^T X)^{-1} } {1 - x_i^T (X^TX)^{-1} x_i}\left( X^T Y - x_i y_i \right)\\ &= \hat{\beta} - (X^T X)^{-1} x_i y_i + \frac{ (X^T X)^{-1} x_i x_i^T \hat{\beta}}{1 - x_i^T (X^TX)^{-1} x_i} - \frac{ (X^T X)^{-1} x_i x_i^T (X^T X)^{-1} x_i y_i }{1 - x_i^T (X^TX)^{-1} x_i}\\ &= \hat{\beta} - (X^T X)^{-1} x_i y_i + \frac{ (X^T X)^{-1} x_i \hat{y}_i}{1 - h_{ii}} - \frac{ (X^T X)^{-1} x_i h_{ii} y_i }{1 - h_{ii}} \\ &= \hat{\beta} - (X^T X)^{-1} x_i y_i(1 + \frac{h_{ii}}{1-h_{ii}}) + \frac{ (X^T X)^{-1} x_i \hat{y}_i}{1 - h_{ii}} \\ &= \hat{\beta} - \frac{(X^T X)^{-1} x_i}{1-h_{ii}}(y_i - \hat{y}_i) \end{align} $$ or $$ \hat{\beta} - \hat{\beta}_{(i)} = \frac{(X^T X)^{-1} x_i}{1-h_{ii}}(y_i - \hat{y}_i). $$

Answer 2

First, try to rewrite the regression equation as

$$\boldsymbol{y}-y_i\boldsymbol{e}_i=\boldsymbol{y}_i=\boldsymbol{X\beta}_i-\boldsymbol{x}_i^T\boldsymbol{\beta}_i\boldsymbol{e}_i,$$

in which $\boldsymbol{e}_i$ is the $i^\text{th}$ unit vector. The sum of squares error is given by

$$E(\boldsymbol{\beta}_i)=\sum_{n=1}^{N}\left[\boldsymbol{y}_i-\boldsymbol{X\beta}_i+\boldsymbol{x}_i^T\boldsymbol{\beta}_i\boldsymbol{e}_i\right]^T\left[\boldsymbol{y}_i-\boldsymbol{X\beta}_i+\boldsymbol{x}_i^T\boldsymbol{\beta}_i\boldsymbol{e}_i\right].$$

The next step is to minimize this expression with respect to $\boldsymbol{\beta}_i$. Can you complete it from here?