What is the intuition behind maximizing the smallest nonzero singular value?

Question

Say I have to maximize the smallest nonzero singular value of a non-square matrix $X$ which is equivalent to maximizing $\lambda_{\min}(X^⊤X)$. What does maximizing the smallest singular value mean? What are some of the applications?

Answer 1

Let's say you are trying to solve a linear system

$$ Ax = b $$

The condition number of a matrix tells how numerically unstable this problem is: if you measure b and you commit a slight measure or rounding error (which you allways do), you will not get the true $b$ but some approximation $b'$. Then if the condition number is large, the error in the solution $x'$ you will get from your linear system solver algorithm can be very large even if the error measuring $b$ was small. This will happen even if your algorithm does not commit any rounding errors!

How does this relate to the singular values? It can be proved the condition number of the matrix is given by

$$ \sigma_{max} / \sigma_{min} $$

where $\sigma_{max}$ and $\sigma_{min}$ are the greatest and smallest singular values of the matrix.

So as $\sigma_{min}$ gets very small, the condition number gets very large and your system gets numerically unstable.

One example where this shows up is in linear regression, where it is very important that the matrix $X^TX$ does not have very small eigenvalues. Indeed, if it does, you say the problem has colinearity and the solutions will be very unstable. Per example, say each column of X represents the answer in a scale 1-10 to a satisfaction survey. Change a little bit of the answers in your survey and the results from the linear regression will be totally different. This means the linear regression model is basically useless in that case.