I'm doing a simple problem in linear algebra. It is clear that I have done something wrong, but I honestly can't see what it is.
let,
$y = Ax$, $y_{ls} = Ax_{ls}$ where A is skinny and full rank, and $x_{ls} = (A^T A)^{-1}A^Ty$ is the standard least squares approximation.
Now, I have tried to compute $y^Ty_{ls}$.
$y^Ty_{ls} = y^TAx_{ls} = x^TA^TA(A^TA)^{-1}A^Ty$
The $A^TA(A^TA)^{-1}$ seems it should cancel to the identity leaving
$x^TA^Ty = y^Ty$.
Clearly this should not be true.
What have I done wrong here?