Linear Least Squares In Matrix Form

Question

if we are given a set of points $\{(x_i,y_i)\}$ and we are looking to fit a straight line $ax+b$ "as close" to the points as possible we are building a set of equations.

One way is to take the partial derivatives with respect to $a,b$ and find the min.

On the matrix form we multiply by $A^T$ when $A$ is entries of $1,x,x^2,..$ why is it smilier to finding the min using derivative?

Answer 1

There is some cookbook about matrix and vector calculus on the internet where we can derive $${\bf x_o}=\min_{\bf x}\{\|{\bf Ax-b}\|_2^2\}$$ by

1. Differentiating the expression $$\|{\bf Ax-b}\|_2^2=({\bf Ax-b})^T({\bf Ax-b})$$ (With respect to $\bf x$)
2. Setting this derivative to zero.
3. Solving for $\bf x$

Doing these steps is a good exercise.