Why doesn't the least squares equation simplify?

Question

I feel like this is very simple question, but I couldn't find the answer to it.

My question is given the least squares equation: $$(A^TA)^{-1}A^T\vec{b}$$ Why can't I simplify it to the following: $$A^{-1}A^{T^{-1}}A^T\vec{b} = A^{-1}I\vec{b} = A^{-1}\vec{b}$$

I know this is incorrect, or else they wouldn't teach it to us, but I'm wondering why. It seems like I've followed the rules. If I take the inverse of the product of 2 matrices, I get the inverse of both with the latter first. If I take the inverse of a matrix and multiply it by the original, I get the identity matrix.

Thank you for your time!

Answer 1

Note that, in the least square method, $A$ is a m-by-n matrix with $m>n$ then we can't simplify the expression $(A^TA)^{-1}A^T\vec{b}$ nor that of the corresponding m-by-m projection matrix $P=A(A^TA)^{-1}A^T$.

Answer 2

The formula $(AB)^{-1}=B^{-1}A^{-1}$ makes sense only when not only $AB$, but also $A$ and $B$ are square matrices. Otherwise, $A$ and $B$ cannot hope for an inverse.

In your settings, $A$ is typically a rectangular matrix. Check this Wikipedia page for more info.