Solution to $AX=XB$ for $3\times3$ rotation matrices.

Question

Given the $3\times3$ rotation matrices $A$ and $B$, find the rotation matrix $X$ that satisfies $$AX=XB.$$

Answer 1

Well, this is equivalent to $A = XBX^{-1}$, so it's only possible if $A$ and $B$ are equivalent. If they are, you can solve this by diagonalizing in $\mathbb{C}$ and finding both base changes $T$, $S$. Then: $A = T^{-1}DT = T^{-1}SBS^{-1}T$ if $B = S^{-1}DS$. And $X = T^{-1}S$.

Edit: This only gives a unitary $X$. Maybe this can be fixed.

Answer 2

If $A,B$ are 3x3 real matrices representing rotations ("3D rotation matrices"), the following are equivalent:

(1) $tr(A) = tr(B)$

(2) There exists real orthogonal $X$ such that $A = XBX^{-1}$.

(3) There exists rotation matrix $X$ such that $A = XBX^{-1}$.

Since rotation matrices are those orthogonal matrices with determinant 1, it's evident that (3) implies (2), and by replacing $X$ by $-X$ if necessary, also (2) implies (3).

Since similar matrices have equal traces, (2) implies (1). It remains only to show (1) implies (2), which we will do in a constructive fashion.

Setting aside the trivial case of zero rotation (identity matrix), every rotation in three dimensions is characterized by an axis of points fixed by the rotation and an angle by which points rotate around that axis. In the case of a special orthogonal linear transformation (rotation represented by a 3x3 matrix), the axis passes through the origin and may be identified by a unit vector $u$ that generates it. If we adopt a convention that positive angles of rotation are counterclockwise about the axis pointed in direction $u$, then changing the sign of the angle is equivalent to changing instead the vector $u$ into $-u$ (and keeping the same angle as before).

As the Wikipedia article points out, if rotation matrix $A$ has angle of rotation $\theta$ about an axis, then the trace of $A$ is $1 + 2 \cos \theta$. For example, by our convention the matrix which represents counterclockwise rotation by $\theta$ about the positive x-axis is:

$$R_x(\theta) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & \cos \theta & -\sin \theta \\ 0 & \sin \theta & \cos \theta \end{pmatrix} $$

and the trace is the sum of diagonal entries.

If $A$ is a rotation matrix with $tr(A) = 1 + 2 \cos \theta$, let $u$ be a unit vector that generates the axis of rotation, i.e. $Au = u$. Extend that to an orthonormal basis $\{u,v,w\}$ for $\mathbb{R}^3$, and obtain an orthogonal matrix $P$ using these basis vectors as columns. Then similar rotation matrix $P^{-1} A P$ is either $R_x(\theta)$ or $R_x(-\theta)$ in the notation above. If the latter, replace $w$ by $-w$, so that:

$$R_x(\theta) = P^{-1} A P$$

Do the same with rotation matrix $B$ also having $tr(B) = 1 + 2 \cos \theta$:

$$R_x(\theta) = Q^{-1} B Q$$

Now $X = P Q^{-1}$ satisfies (2), so proving that (1) implies (2) as desired.