I have to prove that the error vector to least squares solution is orthogonal to the columns of $A$. This means that the error vector $Ax - b$ is in the nullspace of the transpose of $A$. However, I am now stuck.
Here's what I have
\begin{align} x & = (A^TA)^{-1} A^T\cdot b \\ (A^TA)x & = A^T\cdot b \\ (A^T)^{-1}(A^TA)x & = (A^T)^{-1} A^T\cdot b \\ A\cdot x &= \mathbf{b} \\ \end{align}
Can someone give me a hint?
Consider starting with $A^T(Ax-b)$ and show this equals zero, which is another way of saying the error vector is in the null-space of $A^T$. In particular, you'll need to substitute the least-squares solution for $x$ in the above, like this:
$A^T(A[(A^TA)^{-1}A^Tb]-b)$
I hope this helps.