Consider a linear regression $$ y=X_1\beta_1+X_2\beta_2+u $$ to be estimated by least squares. Here, $y$, $X_1$, and $X_2$ are $n\times 1$, $n\times k_1$ and $n\times k_2$. Let $X=(X_1,X_2)$ to be $n\times k$ where $k=k_1+k_2$. Typically, one assumes $X$ is of full column rank.
It is well known that:
- The least squares estimator for $\beta=\begin{pmatrix}\beta_1\\\beta_2\end{pmatrix}$ is given by $\hat\beta\equiv(X'X)^{-1}X'y$. Since $X$ has rank $k$, $X'X$ is invertible so $\hat\beta$ is well-defined.
- The least squares estimator for $\beta_1$ can be written as $\hat\beta_1\equiv(X_1'M_2X_1)^{-1}X_1'M_2y$ where $M_2=I_n-X_2(X_2'X_2)^{-1}X_2'$. (For example, see here).
Since $\hat\beta_1$ is a subvector of $\hat\beta$, $\hat\beta_1$ is obviously well-defined. My questions are:
- How do we see directly that $X_1'M_2X_1$ is invertible from $X$ having full column rank?
- Is $X_1'M_2X_1$ still invertible if we start with assuming $X_1$ has rank $k_1$ only ($M_2$ will have to be redefined as $I_n-X_2(X_2'X_2)^-X_2'$)? Obviously, something beyond $X_1$ having rank $k_1$ is necessary (if $X_2=X_1$, then $X_1'M_2X_1=0$), so what is that "something" please?
$X'X$ is symmetric and is positive semi-definite by definition, which does not yet guarantee invertibility, since $0$ is a possible eigenvalue. To guarantee invertibility one might simply do:
$$X'X = X'X + cI$$
where $c>0$ is a very small number and I is the identity matrix. $X'X + cI$ is now guaranteed to be positive-definite and therefore invertible. For small problems you might now use the Cholesky-Decomposition, for big problems some quadratic optimization algorithm.