How to solve this least square problem effectively?

Question

I want to solve the least square problem, $\min\|Ax-b\|_2$, but the condition number of $A'*A$ is very large, How can I solve this problem effectively?

Answer 1

Hint: Check Moore–Penrose pseudoinverse.

Answer 2

You should try regularization. Several common types are Tikhonov regularization and $\ell^1$ regularization. See this book for a machine learning perspective.

Many common regularizer are of the form $f(x, \lambda)$ where $f(x,\lambda)$ is a non-negative function with tuning parameter $\lambda\geq 0$ to which you solve $\arg\min_x ||Ax-b||_2^2 + f(x,\lambda)$. If $ f=0$, you get the original (unregularized) problem. If $f(x,\lambda) = \lambda ||x||_2$ you get Tikhonov regularization (which can be efficiently implemented via say, the SVD, by applying a certain function to the singular values) (also known as ridge regression). If you have $f(x,\lambda) = \lambda ||x||_1$, this will make solutions of $x$ to be sparser (you may see this under the name LASSO in machine learning-- this is $\ell^1$ regularization).

The Tikhonov regularization problem has a nice form: $x = (A^T A + \lambda I)^{-1} A^T b$, or equivalently, if $A=USV^T$ is the SVD of $A$ with singular values $\{\sigma_i\}$, estimate $x$ by $V D U^T b$ where $D$ has diagonal $\{ \sigma_i/(\sigma_i^2 + \lambda^2)\}$.

The LASSO problem typically doesn't have a nice solution (the orthogonal case in the original paper is the only one I know of)but can easily done with common convex programming packages , e.g, cvx, or with soft thresholding (see the original paper for details, or the papers on ISTA/FISTA (FISTA=Fast Iterative Shrinkage Thresholding Algorithm)).

Answer 3

You can use the SVD Decomposition to solve the Least Squares Problem.
See @dantopa great answer to How does the SVD solve the least squares problem?.

Now, in his answer you see that you need to compute the inverse of the Singular Values Matrix $ S $.
Since you have a system with large conditional number you might be required to invert small number which is problematic.

Another approach would be to threshold small singular values.
Since what's important is the ration between them you might be better threshold (Namely make them zero before inversion) those which are small compared to the largest.

After you clamped those small ones you can invert the matrix and proceed with the SVD solution.

In MATLAB you can do that automatically using the pinv() function and utilizing the tol parameter.

As a side note this will yield a very similar results to Tikhonov Regularization as suggested above (Probably less efficient from computational complexity point of view).
But it is a nice point of view to understand it.

Answer 4

The QR method is the most stable method of solving least square problems.

You need to find the QR factorization of $A$, $A = QR$. Then solution to the least square problem $\min\|Ax-b\|_2$ is given by $x = R^{-1}Q'b$.

For badly conditioned matrices you use the pivoted QR factorization, i.e. $A = QRP$, where $P$ is a permutation matrix, such that diagonal elements of $R$ are sorted decreasingly.