Proof of $(A^TA)^{−1}A^T b=\operatorname{argmin}_x\|Ax-b\|_2$

Question

What is the proof of the following equation?

$$(A^TA)^{−1}A^T b=\operatorname*{argmin}_x\|Ax-b\|_2$$

Answer 1

you can decompose $$b = Ax + (b - Ax).\tag 1 $$ pick the $x$ so that $b - Ax$ is orthogonal to $Ax$ that is $$x^TA^T(b - Ax) = 0 $$ a solution is $$x_0 = (A^TA)^{-1}A^Tb\text{ and } (b-Ax_0) \perp Ax \text{ for any } x .\tag 2$$

to verify the second claim, compute $(Ax)^T(b-Ax_0) = x^T(A^Tb- A^TAx_0) = x^T(A^Tb - A^T b) = 0.$

we also have $$b - Ax = (Ax_0-Ax) +(b-Ax_0) \tag 3 $$

now the phythagoras theorem for a right angle triangle made up of $A(x-x_0), (-Ax_0)$ and the hypotenuse $b- Ax$ shows that $$ |b-Ax|^2 \ge |b-Ax_0|^2 \text{ for all } x.$$ therefore $|b-Ax|$ achieves it minimum at $x = x_0.$