Verifying solution in a linear system $X\beta=y$

Question

Suppose that $X$ is $n\times K$ with full column rank and $y$ $n\times 1$. I understand that if $\beta$ satisfies the system $X\beta=y$, then $\beta=(X'X)^{-1}X'y$ (dimension $k\times 1$). But how do I verify the reverse direction $$ \beta=(X'X)^{-1}X'y\implies X\beta=y? $$ This question is self-contained but it is related to what I asked earlier. Thank you for your help.

Answer 1

We identify the matrix $X$ with the linear map $X:\mathbb R^K \rightarrow \mathbb R^n$, where $\mathbb R^K$ and $\mathbb R^n$ are column spaces. Now, since $X$ has full column rank, we have $K \leq n$. If $K = n$, then $X$ is square and invertible - this is not interesting. So let's assume $K < n$. Then $X \mathbb R^K$ is a strict subset of $\mathbb R^n$, so the equation $K\beta = y$ is not solvable for any given $y \in \mathbb R^n$.

If we are given $y \in \mathbb R^n$ and put $\beta := (X'X)^{-1}X'y$, then we have $X'X\beta = X'y$ or put another way $X'(X\beta - y) = 0$. This implies $(\mathbb R^K)'X'(X\beta - y) = 0$ or equivalently $(X\beta - y) \perp (X \mathbb R^K)$. So we see that $X\beta$ is the orthogonal projection of $y$ onto $X\mathbb R^K$.