I would like to solve for an equation $R_1 R_\mathrm{x} R_2 = R_\mathrm{x}$ where $R_1$, $R_2$, and $R_\mathrm{x}$ are $ 3 \times 3 $ rotation matrices, $R_1$ and $R_2$ are known, and $R_\mathrm{x}$ is unknown. How do I get the solution $R_\mathrm{x}$?
An idea is to represent each rotation matrix as a quaternion, and then compute the elements; how ever it seems to be complicated to me.
If you know a better way, please tell me.
Many thanks!
Note in the below that the inverse of any rotation matrix is its transpose. We can proceed as follows: $$ R_1 R_x R_2 = R_x \iff\\ R_1 R_x R_2 R_x^T = I \iff\\ R_x R_2 R_x^T = R_1^T $$ That is, there will only be a solution matrix $R_x$ if $R_2$ is unitarily similar to $R_1^T$. This is only the case if $R_1$ and $R_2$ are rotations by the same angle about possibly different axes. Assuming this is the case, we can solve this as follows:
First, find an orthogonal $U_1$ so that $$ U_1 R_1 U_1^T = R_\theta := \pmatrix{\cos \theta & -\sin \theta &0\\ \sin \theta & \cos \theta & 0\\ 0 & 0 & 1} $$ (This is a well known problem with known solution algorithms). We can then find an orthogonal $U_2$ so that $$ U_2 R_2 U_2^T = R_\theta := \pmatrix{\cos \theta & -\sin \theta &0\\ \sin \theta & \cos \theta & 0\\ 0 & 0 & 1} $$ It follows that $$ R_1^T = U_1^T R_\theta^T U_1 = U_1^T U_2 R_2 (U_1^T U_2)^T $$ That is, we have the solution $R_x = U_1^T U_2$. So long as we ensure that $\det(U_1)$ and $\det(U_2)$ are positive (which is necessarily possible), we guarantee that $R_x = U_1^T U_2$ is a rotation, as desired.
If you prefer, you may take $$ R_\theta = \pmatrix{e^{i\theta} & 0&0\\0&e^{-i\theta}&0\\0&0&1} $$
For a construction of such a matrix $R_x$, see this post (but replace $R_2$ with $R_2^T$).