Why normalize the eigenvectors when computing SVD and what happens if you do not normalize the eigenvectors?

Question

Suppose that $A = U\Sigma V^\top$, $\Sigma \in \mathbb{R}^{n \times m}$ where $m > n$ (I think this can be relaxed), then a fast way of computing the SVD is as follows:

1. Compute eigenvectors of $AA^\top$, then columns of $U$ are the normalized eigenvectors
2. Compute eigenvectors of $A^\top A$, then rows of $V$ are the normalized eigenvectors
3. $\Sigma$ is equal to the square of $A^\top A$ or $AA^\top$

Why normalize the eigenvectors? What happens if you do not normalize the eigenvectors? Does the unnormalized matrices $U, V$ still allows you to recover $A$?

Answer 1

Another way of phasing the singular value problem is this: suppose we have a linear map $A \colon X \to Y$ between finite-dimensional inner product spaces $V$ and $W$. We'll say that $x_1, \ldots, x_n \in X$ are a singular basis with respect to $A$ if there exist scalars $\lambda_1 \geq \ldots \geq \lambda_n \geq 0$ such that:

1. The $x_i$ are pairwise orthgonal, i.e. we have $\langle x_i, x_j \rangle = 0$ for $i \neq j$,
2. The images of the $v_i$ are pairwise orthogonal, i.e. we have $\langle Ax_i, Ax_j \rangle = 0$ for $i \neq j$, and
3. The operator $A$ "acts with norm $\lambda_i$" along the vector $v_i$, i.e. we have $\lVert Ax_i \rVert = \lambda_i \lVert x_i \rVert$ for all $i$.

Furthermore, you can find that a singular value decomposition exists by getting the $x_i$ as eigenvectors of $A^* A$, and that it is fairly unique (there is some wiggle room for singular vectors with equal singular values). Let us furthermore let $y_1, \ldots, y_m$ be a basis of $Y$ such that $y_i = A x_i$ for all positive singular values $\lambda_i > 0$.

With respect to the bases $(x_i)$ and $(y_i)$, the linear transformation $A$ looks like a diagonal matrix where the $(i, i)$ entry is $1$ if $\lambda_i > 0$, and $0$ otherwise. So while we can define the singular values just fine with unnormalised bases, they don't show up nicely in the matrix factorisation. On the other hand if we normalise the vectors, then both $y_i = A x_i$ and $\lVert A \widehat{x}_i \rVert = \lambda_i$ force that $A \widehat{x}_i = \lambda_i \widehat{y}_i$, and so with respect to these orthonormal bases the operator $A$ looks diagonal, with the singular values down the diagonal.

In summary, you can still do a decomposition $A = U \Sigma V^{-1}$ where $U$ and $V$ have orthogonal columns and $\Sigma$ is diagonal, but the singular values will not appear in $\Sigma$: all you can really tell directly from such a diagonal matrix is the rank of $A$.