Let $M$ be a 3x3 right-handed rotation matrix. I need to find a closed form solution for the right-handed rotation matrix $M'$ where its entries are lie in ${1, 0, -1}$.
An initial idea is to replace each row of $M$ by the unit norm vector that is collinear to its largest component; i.e if the first row is $[0.126, -0.946, -0.299]$, then it should be replaced by $[0, -1, 0]$.
Any thoughts on this? Is there a better approach? What if one of the rotation angles is 45 degrees and the max component is not unique?
To put the problem in its context, this is needed to extract sagittal/axial/coronal slices from MR and CT volumes. The volume files come with rotation matrices that express how the volume is oriented with respect to the standard basis which is the scanner's. Sometimes the slices aren't taken from an orthogonal direction but there's a slight displacement in rotation, so I need to find the closest orthogonal rotation (swap of axis and directions) to know what orthogonal view the volume is taken from. I tried and implemented my solution where I replace each row of my matrix with the unit norm vector on its largest component in absolute value and it works, all my volumes now have an orthogonal rather than oblique rotation.