I have a linear system given by $\mathbf{A} = \mathbf{R}\mathbf{B}$ where $\mathbf{A}$ and $\mathbf{B}$ are 3-by-n matrices and $\mathbf{R}$ is a 3-by-3 orthonormal matrix (i.e., a rotation matrix). Let $\mathbf{M} \equiv \mathbf{A}\mathbf{B}^T$ and its SVD is given by: $$ \mathbf{M} \equiv \mathbf{A}\mathbf{B}^T = \mathbf{U} \mathbf{S} \mathbf{V}^T $$ then I'd like to show that the matrix $\mathbf{R}$ can be written as: $$ \mathbf{R} = \begin{cases} \mathbf{U}\mathbf{V}^T & \text{for} \ det(\mathbf{R}) \geq 0\\ -\mathbf{U}\mathbf{V}^T & \text{for} \ det(\mathbf{R}) < 0\\ \end{cases} $$ Can anybody help me figure out how to prove it?
I do know that using a pseudo-inverse matrix, it follows that $\mathbf{R} = \mathbf{A}\mathbf{B}^T(\mathbf{B}\mathbf{B}^T)^{-1}$ but I'm not sure how it is related to the matrices $\mathbf{U}$ and $\mathbf{V}$.
When $A$, $B$ are not full rank, the choice of $R$ is not unique. For example, if $A= B$ are diagonal matrices with diagonal entries $1, 0, 0$, then $A= RB$ for all $R$ of the form
$$ R = \begin{bmatrix} 1 & & \\ & \cos\theta & \sin\theta \\ &-\sin\theta & \cos\theta\end{bmatrix}.$$
So it is impossible that $R$ is uniquely determined by $U, V$ (unless, for different $R$ you choose a different SVD of $M$).
However, if $n\ge 3$ and $A, B$ both has rank $=3$, one does have uniqueness. The assumption implies that $M= AB^T=RBB^T$ is invertible. Since
$$ M = R (BB^T), \ \ \ M = (UV^T)(V SV^T)$$ give two $QS$ (Q: orthogonal, S: positive semi-definite) decomposition of $A$. By the uniqueness of the QS decomposition (which holds when $M$ is invertible), one has $$ R = UV^T.$$
Remark: It's not clear to me how one has $R = -UV^T$ when the determinant is non-positive. Indeed, since
$$R BB^T = USV^T \Rightarrow \det R \det (BB^T) = \det S \det (UV^T), $$ so $\det R$ and $\det UV^T$ should have the same sign.