I have seen the least squares formula derived like this :
$f(x) = ||Ax-b||_2^2 = \\ (Ax-b)^2 =\\ x^TA^TAx-2b^TAx+b^Tb\\ \nabla f(x) = 2A^TAx -2Ab = 0 => x=(A^TA)^{-1}A^Tb$
I'm trying to derive this myself, but I cant figure out one part from these matrix algebra properties:
$(Ax-b)^2 = (Ax-b)^T(Ax-b) = \\ (Ax-b)^TAx - (Ax-b)^Tb = \\ ((Ax)^T-b^T)Ax - ((Ax)^T-b^T)b = \\ (Ax)^TAx - b^TAx -(Ax)^Tb + b^Tb = \\ x^TA^TAx - b^TAx - x^TA^Tb + b^Tb $
What matrix properties do I need to get : $b^TAx - x^TA^Tb = 2b^TAx$ ?
We have $$(Ax-b)^T\,(Ax-b) = \sum_i (\sum_j A_{ij}x_j - b_i)^2\,, $$ so taking the derivative of the above w.r.t. $x_k$ and setting equal to $0$ we have $$ 0 = \sum_i 2(\sum_j A_{ij}x_j - b_i)(\sum_j A_{ij}\delta_{jk}) = \sum_i 2(\sum_j A_{ij}x_j-b_i)A_{ik} = (A^TAx-A^Tb)_k\,, $$ i.e. $$ x = (A^TA)^{-1}A^T b\,. $$