I am reading online scribe notes on linear regression and I am being quite confused (since did not do linear algebra for a while) on how Pythagorean theorem is being applied here:
First of all I am confused of what "cost" is and then how we get from
$$\arg min_x ||A(x-x') - bb'||_2^2 $$ to
$$\arg min_x ||A(x-x')||_2^2 - ||b'||_2^2 $$ ? E.g I need some intuition and mathematical explanation of what is happening here and how Pythagorean theorem is used in vector space.
Also why $$\arg min_x ||Ax - b||_2 \equiv \arg min_x ||Ax - b||_2^2 $$?
Thank you!
The cost is $\arg\min_x ||Ax-b||_2^2$, and one aims at finding a minimizer $x$.
The "Pythagorean theorem" refers to the fact that
$$||x+y||_2^2 = ||x||_2^2 + ||y||_2^2$$ whenever $x \perp y$.
In general (Hilbert spaces over $\Bbb{C}$), we have
\begin{align} ||Ax-b||_2^2 &= \langle A(x-x') - b', A(x-x') - b' \rangle \\ &= \langle A(x-x'), A(x-x')\rangle - 2\mathop{{\rm Re}}(\langle A(x-x'), b'\rangle) + \langle b',b'\rangle \\ &= ||A(x-x')||_2^2 + 0 + ||b'||_2^2 \tag{$b' \perp A(x-x')$} \\ &= ||A(x-x')||_2^2 + ||b'||_2^2 \end{align}
To address that last problem, observe that for any strictly increasing function $f$, $x < y \iff f(x) < f(y)$ by defintion, so $f(p(x))< f(p(y)) \iff p(x) < p(y)$. Take $f(x) = x^2$ on $[0,\infty)$ to finish the argument.