Deriving projection matrix $QQ^Tb=\hat b$ from normal equation

Question

Suppose normal equation $A^TA\hat x=A^Tb$, with $A$ has linearly dependent columns. Is it possible to directly derive from this normal equation that $QQ^Tb=\hat b$, with $A=QR$? Or we can only prove it from the geometric point of view?

Answer 1

It depends on how you define $\hat{b}$. If you define $\hat{b}$ as the projection of $b$ onto the column space of $A$ then clearly the geometric argument is the way to go. If you define $\hat{b}$ as the "$b$" that the least-square system is really solving for (that is $\hat{b} = A\hat{x}$) then you can derive it algebraically.

Suppose $A = QR$ where $A$ is $m\times n$, $Q$ is an $m \times n$ orthogonal matrix and $R$ is $n \times n$, with $m > n$. Let $\hat{x}$ be the solution to the normal equation $A^TAx = A^Tb$. Define $\hat{b} = A\hat{x}$.

Then

$$ \begin{align*} A^TA\hat{x}& = A^Tb \\ (QR)^T(QR)\hat{x} & = (QR)^Tb \\ R^TR\hat{x} & = R^TQ^Tb \\ R\hat{x}& = Q^Tb \\ QR\hat{x} & = QQ^Tb \\ \hat{b} & = QQ^Tb \end{align*}$$