I condensed down a system of transform matrices to the problem below — in this case I'm dealing with $4 \times 4$ matrices. $A$ and $B$ are known while $X$ is what I'm looking for.
$$X = A X B$$
I'm way out of my element here and wonder if this is solvable and in what direction to go to look for an answer?
for further explanation - my problem comes from having 2 sets of transforms. Each set contains a transform matrix $C_{tn}$ from a coordinate root $C_R$ to a position $C_n$ and a transform matrix $T_{tn}$ from an unknown position $T_R$ to a position $T_n$. I know that the transformation between $C_n$ and $T_n$ is always the same in any measurement but is unknown. So the initial equations I believe are: $$ C_{t1} \cdot X \cdot T_{t1}^{-1} = T_R \\ C_{t2} \cdot X \cdot T_{t2}^{-1} = T_R \\ $$ hence: $$ C_{t1} \cdot X \cdot T_{t1}^{-1} = C_{t2} \cdot X \cdot T_{t2}^{-1} $$ and with $A = C_{t1}^{-1}\cdot C_{t2}$ and $B = T_{t2}^{-1}\cdot T_{t1}$: $$ X = A \cdot X \cdot B $$
This is were I'm stuck and expanding didn't get me much further. I hope this makes sense - I apologize for formatting and terminology.
Vectorizing, one obtains a homogeneous system of $16$ linear equations in $16$ unknowns.
$$\left( {\rm I}_{16} - {\rm B}^\top \otimes{\rm A} \right) \mbox{vec} \left( {\rm X} \right) = {\Bbb 0}_{16}$$
Related: