I can't seem to find anything that would help me with this particular problem. I have a bunch of measurements of a matrix $A$ and corresponding matrix $B$, which I know are related by a third rotation $X$ ($3\times 3$ matrices). How do I arrive at a least squares solution to $X$ where $X$ is constrained to be a rotation?
I thought maybe SVD could be employed somehow, but the approach is not obvious to me.