Proving that $\hat{x} = x$ in least square

Question

I am trying to show that if $Ax=b$ has a unique solution $x$, then the least square solution $\hat{x}$ is the exact one (i.e., $\hat{x} = x$).

My attempt: We know that for $A x = b$ to have an exact unique solution, then $A$ is a full column rank and $b$ is in the column space of $A$.

To minimize the error in the least square, the error $e$ of the projection of $b$ into $C(A)$ has to be orthogonal to the column space of $A$:

$$ e \perp C(A) \Longrightarrow e \in N(A^\top)\\ \Longrightarrow A^\top e = 0\\ A^\top (b-A\hat{x}) = 0 $$

Here I get stuck, because $A^\top e = 0$ might have a special solution (i.e., we don't know if the columns of $A^\top$ are linearly independent).

How can I show that $A^\top e = 0$ has no special solution? (i.e., $e$ must be the zeros vector)? and therefore, $\hat{x}$ is exact? or is there a better way to prove this?

Answer 1

Since $Ax=b$ has a unique solution, $\text{nullity}(A)=0$ and therefore $\text{nullity}(A^{T}A)=0$

$\hspace{.3 in}$since $A$ and $A^{T}A$ have the same nullspace.

Since $Ax=b\implies A^{T}Ax=A^{T}b$, and since $A^{T}A\hat{x}=A^{T}b$, $\;\;A^{T}A(x-\hat{x})=0\implies x-\hat{x}=0$

Answer 2

Id $Ax=b$ has an unique solution then $A$ has number of rows not less than number of columns and it has the full column rank. When number of rows more than number of columns then a non zero $e$ will exist such that $A^Te=0$ but this vector cannot come from $b-Ax$ for any x. Becauseby assumption $b=Ax^*$ and then $A^Te=A^TA(x^*-x)$ can be zero iff $x^*=x$