Confused by the SVD of a real symmetric matrix

Question

For a real symmetric matrix A, it's true that:

$$A A^T = A^T A = A^2$$

And since the right and left singular vectors of $A$ are the eigenvectors of $A^T A$ and $A A^T$ respectively, the right and left singular vectors ought to be identical.

Here's the particular situation that is causing my confusion. Let

$$A = \begin{bmatrix} 4 & 1 & -2 & 2 \\ 1 & 2 & 0 & 1 \\ -2 & 0 & 3 & -2 \\ 2 & 1 & -2 & -1\end{bmatrix}$$

Now when I try to take the SVD of this matrix, the matrices $U$ and $V$, of the left and right singular vectors respectively, are identical. Just as I thought they would be. In particular, for $U$, $\Sigma$, and $V$ I get:

$$U = V = \begin{bmatrix} -0.718 & 0.202 & 0.177 & -0.642 \\ -0.221 & 0.789 & 0.178 & 0.544 \\ 0.557 & 0.580 & -0.288 & -0.520 \\ -0.353 & 0.010 & -0.924 & 0.144 \end{bmatrix}$$

and

$$\Sigma = \begin{bmatrix} 6.845 & 0 & 0 & 0 \\ 0 & 2.269 & 0 & 0 \\ 0 & 0 & 2.198 & 0 \\ 0 & 0 & 0 & 1.084 \end{bmatrix}$$

But if I use this to calculate $U \Sigma V^T$ I don't get $A$. This already confuses me, but to make matters worse, if I take $V$ to be $U$ except with the signs changed on the third column, then it comes out correctly! I feel like I must be misunderstanding something about the SVD or Eigenvectors in general, as I thought that as long as each eigenvector had norm 1, it didn't matter what the sign of it was (i.e. I think about it as more of an eigendirection). Can anyone point out why I'm getting these results?

Answer 1

NOTE that while U == V, the matricies U and V are not symmetric. The definition of the SVD: A = USV' (where the prime indicates the Hermitian (conjugate transpose) of the matrix) - so this is probably your mistake - you probably forgot to apply the transpose - or your math language may have already supplied it for you.

In a lot of the languages with built-in SVD (ie Matlab, Python...) you need to be careful that the SVD function might return U,S,V' (the Hermitian of V)

for example in python you would have U,S,VH = svd(A)

This produces the same U,S,V as you have above - except that it gives the Hermitian of V

Putting it all back together... (in python 'dot' is used for matrix multiplication)

B = dot(U,dot(diag(S),VH))

yielding abs(B-A).max() = 9.992e-16, somewhere around machine epsilon

Answer 2

This is a common mistake when trying to calculate the SVD of symmetric matrices for homework type problems, the root of the issue being the non-uniqueness of eigenvectors.

We know from the SVD of $A=U\Sigma V^T$, that $A^TA = V\Sigma^2 V^T$ (remember that $A$ is symmetric), so find the eigenvectors and eigenvalues of $B=A^TA$. This gives you $V$ and $\Sigma$ (by taking a square root of the eigenvalues), respectively.

Here is the trap. If you were to try to find $U$ by computing the eigenvectors of $AA^T$, you would end up back with $B$ and might conclude that $U=V$. That is wrong, since eigenvectors are not unique. Instead, use the SVD equation, and right multiply by $V$ to get: $AV=U\Sigma=[u_1\sigma_1 \, \ldots \, u_n\sigma_n]$. You already know $A,V$ and $\Sigma$, so normalize the $i^{th}$ column of the product $AV$ by $\sigma_i$ to get $u_i$, the $i^{th}$ column of $U$. This way you get the correct SVD of $A$.

PS: If one or more of the singular values are zero, then the above approach fails. In such cases, you will need to start with those $u_i$'s which correspond to non-zero singular values, and apply a Gram Schmidt procedure with the canonical basis to find the remaining orthonormal vectors to get $U$.

Answer 3

If $A$ is symmetric then $A^TA = A^2$ and the eigenvalues of $A^2$ are the squares of the eigenvalues of $A$, and therefore the singular values are $\sigma=|\lambda|$. The eigenvectors of $A^TA=A^2$ are just the same as those of $A$; call these $v_1,v_2,\ldots,v_n$ and let $V=[v_1\,v_2\,\cdots\,v_n]$. Assume for simplicity that $A$ is non-singular. Then $u = \frac{1}{\sigma} Av$ for each $v$ and corresponding singular value and therefore $u=\frac{\lambda}{|\lambda|} v = sign(\lambda) v$. Let Q be the diagonal matrix containing the numbers $sign(\lambda)$ for each $\lambda$. Then $U=VQ$, and the singular value decomposition is $A=U\Sigma V^T = V (Q \Sigma) V^T$. So, the part $Q\Sigma$ takes care of getting the correct sign for the eigenvalues and it is what you were missing, in other words, $Q\Sigma = diag(\lambda's)$