Minimum norm least squares solution is in the orthogonal complement of kernel

Question

I came across the following statement which I can't prove:

Let $E$ and $F$ be Hilbert spaces and $A:E\to F$ a linear operator. Assume that $\hat{x}$ is the least squares solution to the problem $Ax=y$ closest to an element $x_0\in E$. Then $\hat{x}-x_0\in(\text{Ker}A)^\perp $.

Answer 1

The statement of your problem is not very clear.

An interpretation which makes good sense of the question: You want to minimize $x\in E\mapsto \|y-Ax\|^2\in {\Bbb R}_+$ and if there are several solutions you want to pick the one closest to some $x_0$.

By variation, $x$ is a minimizer iff $A^*A x = A^* y$. Assume that a solution $x_1$ to this equation exists (always true in finite dimensions) then any other solution must lie in the affine subspace $x_1+V$ with $V=\ker A^*A= \ker A$. The one closest to $x_0$ is the orthogonal projection of $x_0$ onto $x_1+V$, i.e. the unique element $\widehat{x}\in x_1+V$ for which $\widehat{x}-x_0\perp V=\ker A$.

Answer 2

Decompose $\hat x = x_0 + \delta + \epsilon$ such that $\delta \in (\rm{Ker} A)^\bot$ and $\epsilon \in \rm{Ker} A$. Suppose that $\epsilon \neq 0$. Then $\|\hat x - x_0\|^2 = \|\delta + \epsilon\|^2 > \|\delta\|^2 = \|(\hat x - \epsilon) - x_0\|^2$, because $\delta$ and $\epsilon$ are orthogonal. Also $A (\hat x - \epsilon) = A\hat x$, because $\epsilon \in \rm{Ker} A$. So $\hat x - \epsilon$ is the same in terms of the least-squares cost, but it is closer to $x_0$ than $\hat x$, which is a contradiction. Therefore $\epsilon = 0$, so $\hat x - x_0 = \delta \in (\rm{Ker} A)^\bot$.

Answer 3

Let $\ x\in\ker A, x\ne0\ $. Then for any complex number $\ z\ $ we have $\ A(\hat{x}+zx)=y\ $, and therefore \begin{align} \|\hat{x}-x_0\|^2&\le\|\hat{x}+z x-x_0\|^2\\ &=\|\hat{x}-x_0\|^2+z\langle x,\hat{x}-x_0\rangle+\overline{z\langle x,\hat{x}-x_0\rangle}+|z|^2\|x\|^2\ , \end{align} from which it follows that $$ 0\le z\langle x,\hat{x}-x_0\rangle+\overline{z\langle x,\hat{x}-x_0\rangle}+|z|^2\|x\|^2\ . $$ Now choosing $\ z=-\frac{\overline{\langle x,\hat{x}-x_0\rangle}}{\|x\|^2}\ $, this becomes $$ 0\le-\frac{|\langle x,\hat{x}-x_0\rangle|^2}{\|x\|^2}\ , $$ which can only be true if $\ \langle x,\hat{x}-x_0\rangle=0\ $. Since $\ x\ $ was an arbitrary non-zero member of $\ \ker A\ $, it follows that $\ \hat{x}-x_0\in(\ker A)^\perp\ $.