How to prove that least squares projection matrix is invariant to constant shifts in $X$?

Question

Suppose you have a matrix $X \in \mathbb{R}^{n \times p}$ and the projection matrix in least squares is defined as:

$$ H = X(X^TX)^{-1}X^T $$

where note that the first column of $X$ are ones.

Now suppose that you define $Y = X - \boldsymbol{1}_{n\times 1} C^T$ where $C \in \mathbb{R}^{p \times 1}$ and $C_1 = 0$. So $Y$ is defined as $X$ shifted by some constant amount, $C_i$ subtracted from the $i$-th column of $X.$

How do we show that

$$ H' = Y(Y^TY)^{-1} Y^T = H $$

Here is my attempt (i'm going to drop the bold on the column vector of 1s): $$ H' = Y(Y^TY)^{-1} Y^T \\ = (X - 1C^T)((X - 1C^T)^T(X - 1C^T))^{-1}(X - 1C^T)^T \\ = (X - 1C^T)((X^T - C1^T)(X - 1C^T))^{-1}(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)(X^T - C1^T) \\ = (X - 1C^T)(X^TX - X^T1C^T - C1^TX + C1^T1C^T)^{-1}(X^T - C1^T) \\ $$

I stopped here because I know this is going to get messy and will probably involve block matrices, and I think there should be an easier solution than what I'm undertaking?

I believe there's an argument that can be made about how $H$ is spanned by the same basis before and after shifting since $X$ is defined to have a column of ones, so any column of $Y$ can be written as the corresponding column of $X$ and the first column of $X$.

Answer 1

The matrix $X$ in this scenario is typically tall and skinny, having many more rows than columns. The matrix $X^TX$ is invertible precisely if the columns of $X$ are linearly independent.

If $u$ is in the column space of $X$ then $u=Xb$ for some column vector $b$ with only as many entries as $X$ has columns, and then a simple algebraic computation shows that $Hu=u.$

But if $u$ is orthogonal to the column space of $X$ then $X^Tu= 0$ and so $Hu=0.$

Thus for any column vector $u$ as "tall" as $X,$ the vector $Hu$ is the orthogonal projection of $u$ onto the column space of $X.$

Therefore, the matrix $H$ depends on $X$ only through the column space of $X.$ If $Y$ is another matrix whose column space is the same as that of $X,$ then $H'=H.$

Answer 2

$\def\o{{\tt1}}$Let $\{e_k\}$ denote the standard vector basis, then the all-ones vector is $$\sum_{k=1}^ne_k=\o$$ The hat matrices for $X$ and $Y$ are $$H_x=X(X^TX)^{-1}X^T=XX^+\\H_y=Y(Y^TY)^{-1}Y^T=YY^+$$ Invoking the first Penrose condition of the pseudoinverse and the fact that $\o$ is the first column of $X$ yields these interesting results $$\eqalign{ H_x\o &= XX^+(\o) = XX^+(Xe_1) = Xe_1 = \o \\ Ye_1 &= (X-\o c^T)e_1 = \o-(c^Te_1)\o = \gamma\o \\ H_y\o &= (YY^+)(\gamma^{-1}Ye_1) = \gamma^{-1}Ye_1 = \o \\ }$$ Assume that $u_x$ lies in the column space of $X$, then for an arbitrary $b$ vector $$\eqalign{ u_x &= Xb \\ H_xu_x &= X(X^TX)^{-1}X^T(Xb) = Xb = u_x \\ }$$ Likewise, if $u_y$ lies in the column space of $Y$ then (utilizing the same $b$ vector) $$\eqalign{ u_y &= Yb \qquad\implies\quad H_yu_y = u_y \\ u_y &= \left(X-\o c^T\right)b \;=\; \big(u_x - \alpha\o\big) \\ }$$ Now compute the action of the hat matrix $H_x$ on $u_y$ $$\eqalign{ H_xu_y &= H_xu_x - \alpha H_x\o \\ &= u_x-\alpha\o \\ &= u_y \\ &= H_yu_y \\ }$$ So the action of $H_x$ on $u_y$ is identical to that of $H_y$

Similarly, the action of $H_y$ on $u_x$ is seen to be identical to that of $H_x$ $$\eqalign{ H_yu_x &= H_y(u_y+\alpha\o) \\ &= u_y + \alpha\o \\ &= u_x \\ &= H_xu_x \\ }$$ Therefore the matrices $H_x$ and $H_y$ are identical.

---

Note that the constraint $$c_1 = c^Te_1 = (1-\gamma) = 0 \quad\implies\quad \gamma=1$$ was not utilized in this analysis. In fact, the above analysis assumes that $$\gamma\ne 0 \quad\implies\quad c_1\ne 1$$

This is basically Michael Hardy's answer, but with more algebra.