I am considering the irreducible representation (IR) of space group 198, which consists of two generators
$$ \left\{C_3\right\}: (x,y,z)\rightarrow(z,x,y)\\ \left\{C_{2x}\big|\frac{1}{2},\frac{1}{2},0\right\}: (x,y,z)\rightarrow(x+\frac{1}{2},-y+\frac{1}{2},-z) $$ There is an additional generator $\left\{C_{2y}\big|0,\frac{1}{2},\frac{1}{2}\right\}$ which satisfies $C_3C_{2x}C_3^{-1}=C_{2y}$, which we will not explicitly consider because given the reps of $C_3,C_{2x}$, we have the rep for $C_{2y}$ automatically.
Now I have two sets of IR $\rho^{1,2}$ which are of the following
$$ \rho^{1}(C_3) = \left( \begin{array}{cccccc} 0 & \frac{1}{2} & 0 & 0 & \frac{\sqrt{3}}{2} & 0 \\ 0 & 0 & \frac{1}{2} & 0 & 0 & \frac{\sqrt{3}}{2} \\ \frac{1}{2} & 0 & 0 & \frac{\sqrt{3}}{2} & 0 & 0 \\ 0 & -\frac{\sqrt{3}}{2} & 0 & 0 & \frac{1}{2} & 0 \\ 0 & 0 & -\frac{\sqrt{3}}{2} & 0 & 0 & \frac{1}{2} \\ -\frac{\sqrt{3}}{2} & 0 & 0 & \frac{1}{2} & 0 & 0 \\ \end{array} \right)\\ \rho^{1}(\left\{C_{2x}\big|\frac{1}{2},\frac{1}{2},0\right\}) = \left( \begin{array}{cccccc} 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & -1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & -1 \\ \end{array} \right) $$ and
$$ \rho^{2}(C_3) = \left( \begin{array}{cccccc} 0 & 0 & -1 & 0 & 0 & 0 \\ -1 & 0 & 0 & 0 & 0 & 0 \\ 0 & -1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & -1 \\ 0 & 0 & 0 & -1 & 0 & 0 \\ 0 & 0 & 0 & 0 & -1 & 0 \\ \end{array} \right)\\ \rho^{2}(\left\{C_{2x}\big|\frac{1}{2},\frac{1}{2},0\right\}) = \left( \begin{array}{cccccc} -1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & -1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \\ \end{array} \right) $$
Question: How to find the unitary matrix $U$ such that $U\rho^{1}(g)U^{-1}=\rho^{2}(g)$ for both $g=C_{3},C_{2x}$?
Such unitary transformation should exist, because the 6-fold IR for SG 198 should be unique. Note also that the characters of the IRs are the same, so I think they should be isomorphic?
Many thanks for the help!
Not an answer, just an extended comment which does not fit in the footnote format for them.
Your 6×6 matrices are manifestly transcribable into tensor products of 2×2 identity matrices I , or Pauli rotations $I/2+i\frac{\sqrt{3}}{2}\sigma_2= \exp (i(\pi/3) \sigma_2 )$, tensored to 3×3 matrices, $$ S\equiv \left( \begin{array}{ccc} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ \end{array} \right), ~~~ S^3=1\!\! 1, \\ T\equiv \left( \begin{array}{ccc} -1 & 0 & 0 \\ 0& 1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right)= S \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0& 1 & 0 \\ 0 & 0 & -1 \\ \end{array} \right) S^2, $$ that is $$ \rho^{1}(C_3) = \exp (i(\pi/3) \sigma_2 ) \otimes S^2, \\ \rho^{1}(\left\{C_{2x}\big|\frac{1}{2},\frac{1}{2},0\right\}) =I\otimes S^2T S ,\\ \rho^{2}(C_3) = -I\otimes S , \\ \rho^{2}(\left\{C_{2x}\big|\frac{1}{2},\frac{1}{2},0\right\}) = I\otimes T~. $$
However, all one can see is that U for $U\rho^{1}(g)U^{-1}=\rho^{2}(g)$ is not a simple tensor product $u\otimes {\mathfrak U}$ of 2×2 and 3×3 matrices, as the respective block traces mismatch in places, e.g. the 2×2 of $C^3$.
While not an answer, the comment might help you with ideas...