If A and A' are approximately the same, are their principal components/SVD very close?

Question

If we have that two matrices $A\approx A'$ within some guaranteed error bound for each term, and $A=U\Sigma V$ is the singular value decomposition for $A$, and $A'=U'\Sigma' V'$ is the SVD for $A'$, is there anything we can say about how close $U$ is to $U'$?

Answer 1

I'm afraid there isn't much to say. An obvious reason is that SVD is not unique, but a deeper reason is that the left singular vectors of $A$ are eigenvectors of $AA^\ast$, and the eigenspaces of a matrix do not necessarily vary continuously with the matrix entries (in contrast, eigenvalues do vary continuously). See this answer on MO for an elegant example.