Understanding the singular value decomposition (SVD)

Question

Please, would someone be so kind and explain what exactly happens when Singular Value Decomposition is applied on a matrix? What are singular values, left singular, and right singular vectors? I know they are matrices of specific form, I know how to calculate it but I cannot understand their meaning.

I have recently been sort of catching up with Linear Algebra and matrix operations. I came across some techniques of matrix decomposition, particularly Singular Value Decomposition and I must admit I am having problem to understand the meaning of SVD.

I read a bit about eigenvalues and eigenvectors only because I was interested in PCA and I came across diagonalizing a covariance matrix which determines its eigenvectors and eigenvalues (to be variances) towards those eigenvectors. I finally understood it but SVD gives me really hard time.

thanks

Answer 1

One geometric interpretation of the singular values of a matrix is the following. Suppose $A$ is an $m\times n$ matrix (real valued, for simplicity). Think of it as a linear transformation $\mathbb R^n \to \mathbb R^m$ in the usual way. Now take the unit sphere $S$ in $\mathbb R^n$. Being a linear transformation, $A$ maps $S$ to an ellipsoid in $\mathbb R^m$. The lengths of the semi-axes of this ellipsoid are precisely the non-zero singular values of $A$. The zero singular values tell us what the dimension of the ellipsoid is going to be: $n$ minus the number of zero singular values.

Answer 2

Maybe it helps to think in terms of linear transformations rather than matrices. Suppose $V$ and $W$ are finite dimensional inner product spaces over $F$ (where $F$ is $\mathbb R$ or $\mathbb C$) and suppose that $T:V \to W$ is a linear transformation. Then, according to the SVD theorem, there exist orthonormal bases $\alpha$ and $\beta$ (bases of $V$ and $W$, respectively) such that $[T]_{\alpha}^{\beta}$ is diagonal.

Trefethen explains a nice geometrical interpretation of the SVD in his book Numerical Linear Algebra.