Intuitively, what is the difference between Eigendecomposition and Singular Value Decomposition?

Question

I'm trying to intuitively understand the difference between SVD and eigendecomposition.

From my understanding, eigendecomposition seeks to describe a linear transformation as a sequence of three basic operations ($P^{-1}DP$) on a vector:

1. Rotation of the coordinate system (change of basis): $P$
2. Independent scaling along each basis vector (of the rotated system): $D$
3. De-rotation of the coordinate system (undo change of basis): $P^{-1}$

But as far as I can see, SVD's goal is to do exactly the same thing, except that resulting decomposition is somehow different.

What, then, is the conceptual difference between the two?

For example:

• Is one of them more general than the other?
• Is either a special case of the other?

Note: I'm specifically looking for an intuitive explanation, not a mathematical one.
Wikipedia is already excellent at explaining the mathematical relationship between the two decompositions ("The right-singular vectors of M are eigenvectors of $M^*M$", for example), but it completely fails to give me any intuitive understanding of what is going on intuitively.

The best explanation I've found so far is this one, which is great, except it doesn't talk about eigendecompositions at all, which leaves me confused as to how SVD is any different from eigendecomposition in its goal.

Answer 1

Intuitively, $SVD$ says for any linear map, there is an orthonormal frame in the domain such that it is first mapped to a different orthonormal frame in the image space, and then the values are scaled.

Eigendecomposition says that there is a basis, it doesn't have to be orthonormal, such that when the matrix is applied, this basis is simply scaled. That is assuming you have $n$ linearly independent eigenvectors of course. In some cases your eigenspaces may have the linear map behave more like upper triangular matrices.

Edit: Consider the difference for a rotation matrix in $\mathbb{R}^2$.

Here, there are no real eigenvalues and this corresponds to there being no choice of basis which under the transformation is simply a scaling. On the other hand, SVD makes a lot of sense here because it says we can take the standard basis in the domain, map it to the rotated version of this basis (thought of as in the image space), and scale everything by 1.

Answer 2

Consider the eigendecomposition $A=P D P^{-1}$ and SVD $A=U \Sigma V^*$. Some key differences are as follows,

• The vectors in the eigendecomposition matrix $P$ are not necessarily orthogonal, so the change of basis isn't a simple rotation. On the other hand, the vectors in the matrices $U$ and $V$ in the SVD are orthonormal, so they do represent rotations (and possibly flips).
• In the SVD, the nondiagonal matrices $U$ and $V$ are not necessairily the inverse of one another. They are usually not related to each other at all. In the eigendecomposition the nondiagonal matrices $P$ and $P^{-1}$ are inverses of each other.
• In the SVD the entries in the diagonal matrix $\Sigma$ are all real and nonnegative. In the eigendecomposition, the entries of $D$ can be any complex number - negative, positive, imaginary, whatever.
• The SVD always exists for any sort of rectangular or square matrix, whereas the eigendecomposition can only exists for square matrices, and even among square matrices sometimes it doesn't exist.