Redundant Vectors in Singular Value Decomposition

Question

Consider a matrix $A\in\mathbb{R}^{2\times3}$. If I perform its Singular Value Decomposition, I'll get the following factorization-

$$A=U\Lambda V^T:U\in\mathbb{R}^{2\times2},~\Lambda\in\mathbb{R}^{2\times3},~V\in\mathbb{R}^{3\times3}$$

The non-zero values of the matrix $\Lambda$ would be its $(1,1)$ and $(2,2)$ elements (assuming a non-singular matrix A). Since the 3rd column of $\Lambda$ is entirely $0$, in the product $U\Lambda V^T$, the 3rd row of $V$ (also 3rd eigen-vector of $A^TA$) would contribute nothing towards the product of $A$. I could in theory replace that vector with anything, and the product $U\Lambda V^T=A$ would remain unchanged.

My question is, then how is the 3rd eigen-vector, in this case, uniquely determined if it contributes nothing to the product and changing it also doesn't change the product (I checked. It doesn't.)? Extending this argument, how are the eigen-vectors corresponding to $0$ eigen-values/singular values uniquely pinned down? Where is that degree of freedom going?

Answer 1

In singular value decomposition, the matrices $U$ and $V$ are orthonormal. That means i.e. that the Columns of $V$ and $U$ are orthonormal bases of the source to and goal space from which the matrix $A$ maps.

The columns in $V$ that correspond to the eigenvectors to eigenvalue $0$ are an orthonormal basis of the $kernel(A)$. Those are the components of directions $x$ which contribute nothing to $Ax$.

The columns in $U$ that correspond to the eigenvectors to eigenvalue $0$ are an orthonormal basis of $image(A)^\perp$. They correspond to the directions $Ax$ cant reach.

Answer 2

My question is, then how is the 3rd eigen-vector, in this case, uniquely determined if it contributes nothing to the product and changing it also doesn't change the product (I checked. It doesn't.)? Extending this argument, how are the eigen-vectors corresponding to 0 eigen-values/singular values uniquely pinned down? Where is that degree of freedom going?

First of all it isn't an eigenvector. There are left and right singular vectors for the singular value decomposition.

Supposing $A \in \mathbb{C}^{m \times n} $ the SVD is

$$A_{m \times n} = U_{m \times m} \Sigma_{m \times n} V_{n \times n}^{*} $$

Notably, if the rank of the matrix is only $r$ then we only have $r$ singular values. We can truncate the singular value decomposition. It is as follows $$A_{m \times n} = U_{m \times r} \Sigma_{r \times r} V_{r \times n}^{*} $$

This can be visually seen as the following.

This follows from how the SVD is obtained. The singular values are the square roots of the eigenvalues of the covariance matrix $A^{T}A $ or $AA^{T}$

$$AA^{*} = (U\Sigma V^{*}) (U\Sigma V^{*})^{*} $$ $$AA^{*} = U\Sigma V^{*} V\Sigma^{*} U^{*} $$

$V^{*}V = VV^{*} = I $

$$AA^{*} = U\Sigma \Sigma^{*} U^{*} $$

$ \Sigma \Sigma^{*} = \Sigma^{2} = \Lambda $

$$AA^{*} = U \Lambda U^{*} $$

The left singular vectors

other way, now

$$A^{*}A = (U\Sigma V^{*})^{*} (U\Sigma V^{*}) $$ $$A^{*}A = V\Sigma^{*} U^{*} U\Sigma V^{*} $$

$$UU^{*} = U^{*}U =I $$

$$A^{*}A = V\Sigma^{*}\Sigma V^{*} $$ $$ \Sigma^{*}\Sigma = \Lambda $$ $$A^{*}A = V \Lambda V^{*} $$ Giving us the right singular vectors

Now the same idea follows through for truncated eigenvalue decomp.

$$ A_{m \times m} = V_{m \times m} \Lambda_{m \times m} V_{m \times m}^{*} $$

if the rank of the matrix is $r$ we have 0's there

$$ A_{m \times m} = V_{m \times r} \Lambda_{r \times r} V_{r \times m}^{*} $$

In a machine, they'd be very close to machine precision if the rank if $r$. You have two choices, you can get the full decomposition and you'd have singular values that are around $1e-16$ or choose a truncated one.