I’ve been trying to understand the details of orthogonal projections within the context of linear regression models, and I’ve come across the following scenario:
Given a linear regression model $Y=\beta_0+\beta_1X_1+\beta_2X_2+\epsilon$ where $X=\begin{pmatrix} 1 & x_{1,1} & x_{1,2} \\ 1 & x_{2,1} & x_{2,2} \\ 1 & x_{3,1} & x_{3,2} \end{pmatrix} $ and $ \beta = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{pmatrix} $, consider the projection matrix onto the column space of $X$, $H=X(X’X)^{-1}X’$. We define matrix $\tilde X $ as a matrix composed of some columns of $X$, and $\tilde H$ as the projection matrix onto the column space of $\tilde X$. i.e. without loss of generality $\tilde X=\begin{pmatrix} 1 & x_{1,1} \\ 1 & x_{2,1} \\ 1 & x_{3,1} \end{pmatrix} $ and $\tilde H=\tilde X(\tilde X’\tilde X)^{-1}\tilde X’ $. Formally prove that $\tilde H HY = \tilde HY$.
We’re additionally given a hint, to first prove that $H\tilde H= \tilde H$.
I have shown that $\tilde H HY = \tilde HY$ by a simple approach:
The projection of $Y$ onto $span(X)$ is given by $\hat Y= HY$. So $e=Y-\hat Y \in span(X)^{\perp}$ (due to the definition of an orthogonal projection, or one can easily show that $H(Y-HY)=0$ because $H$ is idempotent).
Then $\tilde H HY -\tilde HY = \tilde H (HY-Y) = \tilde H(\hat Y-Y) = -\tilde He $
Since $span(\tilde X) \subseteq span(X)$ it implies that for every $v\in span(\tilde X): e \perp v \implies \tilde He= 0$.
Thus, $\tilde H HY -\tilde HY = - \tilde He = 0 \implies \tilde HH = \tilde H$.
I don’t believe this was the author’s intention, due to the explicit definitions of the model and design matrix, as well as the given hint. Perhaps we should rigorously prove that the product of the projection matrices is commutative.
The fact that $H\tilde H= \tilde H$ is much more intuitive to me, and I’ve proved it in a similar fashion, considering properties of orthogonal projections and the fact that $span(\tilde X) \subseteq span(X)$.
I would love to hear your thoughts on this, how do I rigorously prove that $\tilde H HY = \tilde HY$, given that $H\tilde H= \tilde H$?