Help with Singular Value Decomposition

Question

I have a matrix $A$, $n\times m$. With singular value decomposition we have $A=U\Sigma V^{T}$

Why the singular values of matrix $\Sigma$ are in descending order?

Answer 1

If you permute the columns of $U$ and apply the same permutation to the columns of $V$ (or rows of $V^t$), and permute the corresponding entries of $\Sigma$, you'll get an equally good decomposition with the singular values NOT in decreasing order. But the usual definition requires decreasing order as a way to make the SVD unique, at least for matrices with distinct singular values.

By the way, when there are repeated singular values, there will be many possible SVDs. For instance, if $A = I$, then $A = Q I Q^t$ for any orthgonal matrix $Q$.

Post-Comment Addition

By the way, even with decreasing singular values, you don't get true uniqueness: If you negate any row of $U$ and the corresponding column of $V$, the decomposition remains correct.

To be clear about my first claim:

Let $q = \sqrt{2}/2, c = \frac{1}{2} = \cos{\frac{\pi}{3}}, S = \frac{\sqrt{3}}{2} = \sin{\frac{\pi}{3}}$. Then for the matrix $$ M = \begin{bmatrix} -q && -q \\ -q && q \end{bmatrix} \begin{bmatrix} 2q && 0 \\ 0 && q \end{bmatrix} \begin{bmatrix} -c && s \\ s && c \end{bmatrix}, $$ we also have $$ M = \begin{bmatrix} -q && -q \\ q && -q \end{bmatrix} \begin{bmatrix} q && 0 \\ 0 && 2q \end{bmatrix} \begin{bmatrix} s && c \\ -c && s \end{bmatrix}, $$

So if you allow yourself to permute the singular values, then the decomposition can change.

If you fix the order of the singular values, even if this order is not decreasing, and if all singular values are distinct, then the SVD is unique up to negating any column of $U$ and the corresponding column of $V$ (or row of $V^t$).

Answer 2

People like uniqueness in their decompositions. There are usually some ambiguities in them, but we try to reduce them as far as posible. For the SVD, we know that the singular values are always real-valued (in fact also nonnegative) which allows us to order them. If they are distinct, this eliminates permutation ambiguities among the singular vectors/values.

For the eigenvalue decomposition we cannot do this, since the eigenvalues are in general complex which forbids a unique ordering.

The SVD is still not fully unique though, you can for instance absorb a minus sign into a column of $U$ and the matching columns of $V$ without changing the result. Not much we can do about that.