Projection Matrix Expressed as QR Identity

Question

Given the projection matrix in least squares, $H=X(X^TX)^{-1}X^T$, let $X = QR$ as in QR decomposition, and express $H$ in terms of $Q$.

No matter what I do, I either end with $H = I$, or a term using $R$ that I cannot reduce to just $Q$.

Answer 1

Note that $X^T X = R^T Q^T Q R = R^T R$ (as $Q^T Q = I$).

• Assume $R$ is invertible. Then you have $$ H = Q R \left( R^T R \right)^{-1} R^T Q^T = Q R R^{-1} R^{-T} R^T Q^T = Q Q^T $$

• If $R$ is not invertible then $X$ is not full rank and $X^T X$ is not invertible.

Answer 2

In your formula for OLS solution $H$, $H=X(X^TX)^{-1}X^T$, you are assuming that $X$ has a full column-rank, which implies that $R^T R$ is invertible, i.e. $(R^T R)^{-1}$ exists. However, if $X$ is not a square matrix, $X^{-1}$ does not exist. You can define its pseudoinverse though. There are many types of pseudoinverse definitions (e.g. which of 4 conditions in Moore–Penrose inverse definition are satisfied) as generalization of the normal matrix inverse, and the most popular/useful one is Moore–Penrose inverse (see, for example, https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse).

Let's do your calculation using $QR$ decomposition

$H=X(X^TX)^{-1}X^T=QR[(QR)^T(QR)]^{-1}(QR)^T$=$QR[R^TQ^TQR]^{-1}R^TQ^T$=$QR[R^TR]^{-1}R^TQ^T$.

Note that $[R^TR]^{-1} \ne R^{-1}R^{-T}$ if $X$ is not a square matrix, because the right-hand-side parts are not defined.

Assume $X$ is an $m\times n$ matrix with $m\ge n$.

If $X$ is a square invertible matrix, you will have $H=I$.

If $X$ is not square ($m>n$) and has a full column rank, then using Moore-Penrose inverse, it will be reduced to $H=QUKU^TQ^T$, where $Q$, $U$ and $K$ are all $m\times m$ matrices, $Q$ and $U$ are Hermitian, $K=(k_{ij})$ is a diagonal matrix, $k_{ii}=1$ for $i=1,...,n$, and $k_{ij}=0$ for other entries.

In other words, we can find a Hermitian matrix $U$ and diagonal matrix $K$ like above such that $R[R^TR]^{-1}R^T=UKU^T$. Its proof is very straightforward using SVD format of Moore-Penrose inverse. Again, $K=I$ will be the identity matrix if $m=n$.