I am trying to understand Singular Value Decomposition from a intuitive point of view. As we know by applying SVD to any matrix, we get these three matrices as U, Sigma, V. And a matrix multiplication can be interpreted as a combination of rotation and scaling. Here U & V corresponds to the rotation performed by that matrix and Sigma the scaling factor.
My question is, why do we get or need two rotation matrices or unitary matrices (U & V), instead of one?
On
Let $A = V\Sigma U^t$ over $\mathbb R$.
TL;DR: 1) Matrix $U$ defines the orthogonal directions along which the scaling will take place. This is one rotation whose angle is completely defined by the start direction of the scaling. 2) Matrix $V$ defines where these stretch directions end up afterwards.
$U$ rotates the input object (by an angle $\theta$ if $x\in\mathbb R^2$) such that the point $x$ to be stretched $\sigma_1$ will be along the $x_1$ direction (analogously for $x_i$).
$V$ rotates this (by an angle $\phi$ if V is $2\times 2$) to now position the final object arbitarily as a rotated version.
If you only had $U$, you would be able to stretch the object at arbitrary directions with $U\Sigma U^t$ or to rotate the object to the stretch direction with $\Sigma U^t$ but not rotate it arbitrarily.
The Singular Value Decomposition theorem states that if $V$ and $W$ are complex inner product spaces, and $L:V \to W$ is a linear map, then I can find an orthonormal basis $v_1,v_2,...,v_n$ for $V$ and an orthonormal basis $(w_1,w_2,...,w_m)$ of $W$ such that $L(v_i) = \sigma_i w_i$.
When you translate this into a statement about the matrix of the linear map, you need to apply a "change of basis" matrix to the domain and the codomain. These are unitary since the basis is orthonormal. This is where your two unitary bases come from. The matrix for $L$ is diagonal WRT these bases, so that is where the diagonal matrix comes from.