I was reading this Wikipedia page and I understood the math until the $\vec{y} = \mathbf{X}\vec{\beta}+\vec{\varepsilon}$ equation (I understand that equation). After that, I cannot follow the math. Also, I currently am only concerned about exact fit polynomials, so all matrices that will be inverted will be square.
Can anybody explain the derivation? Thanks!
The error term is $y-Xb$. What follows is the standard procedure for minimizing this error.
$$\|\epsilon\|^2=\|y-Xb\|^2=(y-Xb)^\top(y-Xb)=y^\top y-2b^\top X^\top y+b^\top X^\top Xb$$ Differentiating wrt $b$ gives the minimum, $$-2X^\top y+2X^\top Xb=0$$ hence $b=(X^\top X)^{-1}X^\top y$.