X is a $n \times p$ matrix. If we let $X_{(-i)}$ denote the $(n-1) \times p$ matrix obtained by deleting the $i^\mathrm{th}$ row of $X$, and this matrix is also of full rank $p$, how do I show:
$X_{(-i)}^\top X_{(-i)} = X^\top X - x_ix_i^\top$
Appreciate the help! Thanks!
You can write down your $X^TX$ as in an outer product form such that
$X^TX=\begin{pmatrix}x_1 & x_2 & \cdots & x_i & \cdots x_n\end{pmatrix}\begin{pmatrix}x_1^T \\ x_2^T \\ \vdots \\ x_i^T \\ \vdots \\ x_1^T\end{pmatrix}=\sum_{k=1}^nx_kx_k^T$,
and you can do the same thing with $X_{(-i)}$ resulting into
$X^TX=\begin{pmatrix}x_1 & x_2 & \cdots & x_{i-1} & x_{i+1}& \cdots x_n\end{pmatrix}\begin{pmatrix}x_1^T \\ x_2^T \\ \vdots \\ x_{i-1}^T \\ x_{i+1}^T \\ \vdots \\ x_1^T\end{pmatrix}=\sum_{k=1}^{i-1}x_kx_k^T+\sum_{k=i+1}^{n}x_kx_k^T$
where $x_i$ represents the rows of the matrix written down as column vectors. From this points it is obvious that the second sum just lacks from the $x_ix_i^T$ factor comparing it to the first sum. Consequently
$X_{(-i)}^TX_{(-i)}=X^TX-x_ix_i^T.$