I am currently studying this answer on the Moore-Penrose pseudoinverse and the Euclidean norm by the user "Etienne dM". The first point of their answer proceeds as follows:
Let $x$ be $A^+y$.
- Let me begin by the second point. For all $z$, we have: \begin{align} \lVert Az-b \rVert_2^2 &= \lVert Ax-b \rVert_2^2 + \lVert A(z-x) \rVert_2^2 + 2 (z-x)^TA^T(Ax-b)\\ & \geq \lVert Ax-b \rVert_2^2 + 2 (z-x)^TA^T(Ax-b) \end{align} Moreover, because $(AA^+)^T = AA^+$, $$ A^T(Ax-b) = ((AA^+)A)^Tb - A^Tb = 0$$ Thus, we prove that for all $z$, $\rVert Az-b \lVert_2^2 \geq\rVert Ax-b \lVert_2^2$, that is to say $A^+b$ is as close as possible to $y$ in term of the Euclidian norm $\lVert Ax-b\rVert_2$.
I realise that $x = A^+ y$, but I don't understand how any of this implies that "$A^+ b$ is as close as possible to $y$ in terms of the Euclidean norm $\lVert Ax-b\rVert_2$". I would greatly appreciate it if people would please take the time to explain this to me.
We assume that $(A^TA + \alpha I)^{-1}A^T$ indeed has a limit as $\alpha \to 0^+$. Let $x$ be given by $$ x = A^+ y = \lim_{\alpha \to 0^+ }[(A^TA + \alpha I)^{-1}A^T y]. $$
Consider any $\alpha > 0$. We note that $x_\alpha = (A^TA + \alpha I)^{-1}A^Ty$ is the unique solution to the system $$ (A^TA + \alpha I)x_{\alpha} = A^Ty, $$ and that $x_{\alpha} \to x$ as $\alpha \to 0^+$. It follows that $$ \|A^Ty - A^TAx\| = \lim_{\alpha \to 0^+}\|A^T y - A^TAx_{\alpha}\| = \lim_{\alpha \to 0^+}\|\alpha x_{\alpha}\| \\ \qquad \qquad = \lim_{\alpha \to 0^+} \alpha \|x_{\alpha}\| = \lim_{\alpha \to 0^+} \alpha \|x\| = 0. $$ So, we indeed have $A^TAx = A^Ty$, which means that $x$ is a least-squares solution to $Ax = y$, which is what we wanted.