Show that $X^T_{(-i)} X_{(-i)} = X^TX - x_ix^T_i$.

74 Views Asked by At

$X$ is an $n$ x $p$ matrix with rank $p$, and $X_{(-i)}$ is an $(n-1)$ x $p$ matrix that is the same as $X$ but with the $i$th row removed. $x_i^T$ represents this row.

I've tested this in MATLAB and it makes sense: removing a row from the matrix and taking $X^T_{(-i)} X_{(-i)}$ is the same as using the full matrix and taking away $x_ix^T_i$. I've tried to prove this using a general case of $X$, but it seems more tedious to prove it this way and I feel like there is an easier method, possibly using the fact that $X$ is of full rank.

1

There are 1 best solutions below

0
On BEST ANSWER

(Edit: added the zero-blocks explicitly, thus making the computations look neater.)

Consider the $n$-order identity matrix $$I_n =\begin{bmatrix} I_{i-1}& {\bf 0}_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}^{\rm T}_{i-1}& 1& {\bf 0}^{\rm T}_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{n-i}& I_{n-i} \end{bmatrix},$$ where the $i$th row (let us call it $e^{(n)\rm T}_i$) and column are somewhat highlighted, and the bolded zeros are blocks of zeros (columns or blocks, according to the subscripts). Removing a row from $I_n$ and then multiplying it from the left to another matrix $M$ is the same as deleting the corresponding row in $M$, thus we can express $X_{(-i)}$ as $$X_{(-i)} :=\begin{bmatrix} I_{i-1}& {\bf 0}_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{n-i}& I_{n-i} \end{bmatrix} X.$$ Taking the transpose: $$X^{\rm T}_{(-i)} =X^{\rm T} \begin{bmatrix} I_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}^{\rm T}_{i-1}& {\bf 0}^{\rm T}_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)}& I_{n-i} \end{bmatrix}.$$ Multiplying $X^{\rm T}_{(-i)} X_{(-i)}$ gives: $$X^{\rm T} \begin{bmatrix} I_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}^{\rm T}_{i-1}& {\bf 0}^{\rm T}_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)}& I_{n-i} \end{bmatrix} \begin{bmatrix} I_{i-1}& {\bf 0}_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{n-i}& I_{n-i} \end{bmatrix} X\\ =X^{\rm T} \begin{bmatrix} I_{i-1} I_{i-1} +{\bf 0}_{(i-1)\times(n-i)} {\bf 0}_{(n-i)\times(i-1)}& I_{i-1} {\bf 0}_{i-1} +{\bf 0}_{(i-1)\times(n-i)} {\bf 0}_{n-i}& I_{i-1} {\bf 0}_{(i-1)\times(n-i)} +{\bf 0}_{(i-1)\times(n-i)} I_{n-i}\\ {\bf 0}^{\rm T}_{i-1} I_{i-1} +{\bf 0}^{\rm T}_{n-i} {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}^{\rm T}_{i-1} {\bf 0}_{i-1} +{\bf 0}^{\rm T}_{n-i} {\bf 0}_{n-i}& {\bf 0}^{\rm T}_{i-1} {\bf 0}_{(i-1)\times(n-i)} +{\bf 0}^{\rm T}_{n-i} I_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)} I_{i-1} +I_{n-i} {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{(n-i)\times(i-1)} {\bf 0}_{i-1} +I_{n-i} {\bf 0}_{n-i}& {\bf 0}_{(n-i)\times(i-1)} {\bf 0}_{(i-1)\times(n-i)} +I_{n-i} I_{n-i} \end{bmatrix} X$$ $$\begin{align} &= X^{\rm T} \begin{bmatrix} I_{i-1} +{\bf 0}_{(i-1)\times(i-1)}& {\bf 0}_{i-1} +{\bf 0}_{i-1}& {\bf 0}_{(i-1)\times(n-i)} +{\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}^{\rm T}_{i-1} +{\bf 0}^{\rm T}_{i-1}& 0 +0& {\bf 0}^{\rm T}_{n-i} +{\bf 0}^{\rm T}_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)} +{\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{n-i} +{\bf 0}_{n-i}& {\bf 0}_{(n-i)\times(n-i)} +I_{n-i} \end{bmatrix} X\\ &= X^{\rm T} \begin{bmatrix} I_{i-1}& {\bf 0}_{i-1}& {\bf 0}_{(i-1)\times(n-i)}\\ {\bf 0}^{\rm T}_{i-1}& 0& {\bf 0}^{\rm T}_{n-i}\\ {\bf 0}_{(n-i)\times(i-1)}& {\bf 0}_{n-i}& I_{n-i} \end{bmatrix} X. \end{align}$$

That thing between $X^{\rm T}$ and $X$ is a $n$-order identity matrix but with a zero in the $(i,i)$ entry; express it as $I_n$ minus a $n$-order matrix with zeros everywhere but a one in the $(i,i)$ entry: $$\begin{align} X^{\rm T}_{(-i)} X_{(-i)} &= X^{\rm T} (I_n -e^{(n)}_i e^{(n)\rm T}_i) X\\ &= X^{\rm T} I_n X -X^{\rm T} e^{(n)}_i e^{(n)\rm T}_i X\\ &= X^{\rm T} X -(e^{(n)\rm T}_i X)^{\rm T} e^{(n)\rm T}_i X. \end{align}$$ That thing $e^{(n)\rm T}_i X$ represents the $i$th row of $X$, that is, $x^{\rm T}_i$. $$X^{\rm T}_{(-i)} X_{(-i)} = X^{\rm T} X -x_i x^{\rm T}_i.$$ I hope this way is easier and not so tedious.