Least squares invariant under transformation?

533 Views Asked by At

Suppose that $\hat{x}$ is the least squares solution of $Ax = b$. That is, $\min{|r|}, r = Ax - b$ with respect to $x$.

Is $\hat{x}$ also the least squares solution of $JAx = Jb$? I suspect so, but I'm not sure how to prove it. If $J$ is invertible, it's easy enough to prove as you can just cancel it out from both sides. But what if it's not invertible?

1

There are 1 best solutions below

1
On BEST ANSWER

The answer is NO. The vector $\hat{x}$ won't generally be a solution of the transformed least squares problem unless, e.g., $J$ is identity or orthogonal or $Ax=b$ is consistent.

Also, note that if $J$ is nonsingular, then of course, you can "cancel out" $J$ somewhat but it won't give you the same least squares solution! Recall that you do not solve an "ordinary" consistent system but a least squares problem whose solution depends on what you minimize and in which norm. Which leads to the following remark:

Multiplying the system from left changes the norm, so instead of minimizing $\|b-Ax\|$, you minimize $\|J(b-Ax)\|$ which can be something completely different. The two problems can have different solutions.

Example: Consider $$ A=\begin{bmatrix}1&1\\1&0\\1&0\end{bmatrix}, \quad b=\begin{bmatrix}1\\0\\1\end{bmatrix}, \quad J=\begin{bmatrix}1&1&0\\0&1&0\\0&0&1\end{bmatrix}. $$ The solutions of the problems $Ax\approx b$ and $JAy\approx Jb$ are $$ \hat{x} = \begin{bmatrix}1/2\\1/2\end{bmatrix}\quad\text{and}\quad \hat{y} = \begin{bmatrix}1/2\\0\end{bmatrix}. $$