Subgroups and invariants in a unitary group $U(3)$ mixing multiplets

Question

This is related to the post, but an enriched version of the problem. Now we require the richer form of $P_{x1},P_{x2},P_{x3},P_{y1},P_{y2},P_{y3},P_{z1},P_{z2},P_{z3}$.

Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of $U(3)$. Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrix.

Can we find some subgroup of Lie group, $$k \in K \subset G= U(3) $$ such that

$$ k^T \{ \pm P_{x1},\pm P_{x2},\pm P_{x3},\pm P_{y1},\pm P_{y2},\pm P_{y3},\pm P_{z1},\pm P_{z2},\pm P_{z3} \} k =\{ \pm P_{x1},\pm P_{x2},\pm P_{x3},\pm P_{y1},\pm P_{y2},\pm P_{y3},\pm P_{z1},\pm P_{z2},\pm P_{z3}\}. $$ This means that set $\{ \pm P_{x1},\pm P_{x2},\pm P_{x3},\pm P_{y1},\pm P_{y2},\pm P_{y3},\pm P_{z1},\pm P_{z2},\pm P_{z3}\}$ is invariant under the transformation by $k$. Here $k^T$ is the transpose of $k$. What is the full subset (or subgroup) of $K$?

Here we define:

$$ P_{x1} = \left( \begin{array}{ccc} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right) ,\;\;\;\; P_{y1} = \left( \begin{array}{ccc} -1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_{z1} = \left( \begin{array}{ccc} -i & 0 & 0 \\ 0 & -i & 0 \\ 0 & 0 & 0 \\ \end{array} \right).$$ $$ P_{x2} = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \\ \end{array} \right) ,\;\;\;\; P_{y2} = \left( \begin{array}{ccc} -1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right),\;\;\;\; P_{z2} = \left( \begin{array}{ccc} -i & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -i \\ \end{array} \right).$$ $$ P_{x3} = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1& 0 \\ \end{array} \right),\;\;\;\; P_{y3} = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 1 \\ \end{array} \right),\;\;\;\; P_{z2} = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & -i & 0 \\ 0 & 0 & -i \\ \end{array} \right).$$

This means that $k^T P_a k= \pm P_b$ which may transform $a$ to a different value $b$, where $a,b \in \{x1,x2,x3,y1,y2,y3,z1,z2,z3 \}$. But overall the full set $ \{ \pm P_{x1},\pm P_{x2},\pm P_{x3},\pm P_{y1},\pm P_{y2},\pm P_{y3},\pm P_{z1},\pm P_{z2},\pm P_{z3}\}$ is invariant under the transformation by $k$.

There must be a trivial element $k=$ the rank-3 identity matrix. But what else can it allow?

How could we determine the complete $K$? What is the order of $K$? What is $K$ isomorphic to? [A more familiar group like dihedral, symmetry groups, cyclic, etc?]

Answer 1

Denote the set of the matrices by $\mathcal{P}=\{\pm P_{xi},\pm P_{yi},\pm P_{zi} \mid 1\leq i\leq 3\}$.

Let $K=\{k \in U(3) \mid k^TPk\in\mathcal{P}\text{ for all $P\in\mathcal{P}$}\}$, the set of matrices under which $\mathcal{P}$ is invariant.

Claim 1. There is no $k\in K$ such that $k^TP_{x1}k=\pm P_{z1}$.

Proof. Denote by $k=(k_{ij})$. Since $k$ is unitary, $k^TP_{x1}k=\pm P_{z1}$ implies $$ P_{x1}k = \begin{pmatrix} k_{21} & k_{22} & k_{23} \\ k_{11} & k_{12} & k_{13} \\ 0 & 0 & 0 \end{pmatrix} = \pm \begin{pmatrix} -i\bar k_{11} & -i\bar k_{12} & 0 \\ -i\bar k_{21} & -i\bar k_{22} & 0 \\ -i\bar k_{31} & -i\bar k_{32} & 0 \end{pmatrix} = \pm \bar kP_{z1} $$ and thus $k_{13}=k_{23}=k_{31}=k_{32}=0$.

We now have $k=\begin{pmatrix} k_{11} & k_{12} & 0 \\ k_{21} & k_{22} & 0 \\ 0 & 0 & k_{33} \end{pmatrix}\in U(2)\times U(1)$. Notice that $k_{33}\neq 0$.

Then $k^TP_{xi}k$ have the matrix form as follows: $$ \begin{align*} k^TP_{x1}k &= \begin{pmatrix} 2k_{11}k_{21} & k_{11}k_{22}+k_{12}k_{21} & 0 \\ k_{11}k_{22}+k_{12}k_{21} & 2k_{12}k_{22} & 0 \\ 0 & 0 & 0 \end{pmatrix} = \pm \begin{pmatrix} -i & 0 & 0 \\ 0 & -i & 0 \\ 0 & 0 & 0 \end{pmatrix} = \pm P_{z1} \\ k^TP_{x2}k &= \begin{pmatrix} 0 & 0 & k_{11}k_{33} \\ 0 & 0 & k_{12}k_{33} \\ k_{11}k_{33} & k_{12}k_{33} & 0 \end{pmatrix} = \pm P_{x2} \text{ or } \pm P_{x3} \end{align*} $$

From the first equation that is our assumption, we know that all $k_{11}$, $k_{12}$, $k_{21}$, $k_{22}$ are not zero.

Remember that $k_{33}\neq0$. If $k^TP_{x2}k=\pm P_{x2}$, then $k_{12}=0$, which contradicts to $k_{12}\neq0$. Similarly, if $k^TP_{x2}k=\pm P_{x3}$, then $k_{11}=0$, which contradicts to $k_{11}\neq0$.

Claim 2. There is no $k\in K$ such that $k^TP_{x1}k=\pm P_{y1}$.

Proof. Completely the same as Claim 1.

Set $Q=\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix} \in U(3)$. Then $Q^T=Q^{-1}=Q^2$ and $$ \begin{gather*} QP_{x1}Q^T=P_{x2}, \quad QP_{x2}Q^T=P_{x3}, \quad QP_{x3}Q^T=P_{x1} \\ QP_{y1}Q^T=-P_{y2}, \quad QP_{y2}Q^T=-P_{y3}, \quad QP_{y3}Q^T=P_{y1} \\ QP_{z1}Q^T=P_{z2}, \quad QP_{z2}Q^T=P_{z3}, \quad QP_{z3}Q^T=P_{z1} \\ \end{gather*} $$ It implies that if $k\in K$ then $kQ\in K$ and $kQ^T\in K$.

Claim 3. There is no $k\in K$ such that $k^TP_{x1}k=\pm P_{z2}$ or $k^TP_{x1}k=\pm P_{z3}$.

Proof. If $k^TP_{x1}k=\pm P_{z2}$, then $(kQ)^TP_{x1}(kQ)=Q^T(k^TP_{x1}k)Q=\pm Q^TP_{z2}Q=\pm P_{z1}$, which contradicts to Claim 1.

Claim 4. There is no $k\in K$ such that $k^TP_{x1}k=\pm P_{y2}$ or $k^TP_{x1}k=\pm P_{y3}$.

Proof. Completely the same as Claim 3.

Now we have a conclusion that if $k\in K$, then $k^TP_{x1}k=\pm P_{xi}$ for some $1\leq i\leq 3$. Moreover, we already know the solution in this case, that is,

The subgroup $K$ of $U(3)$ containing invariant matrices are isomorphic to the finite group $$ \mathbb{Z}_4\times S_4 \simeq\langle i\rangle\times D(2,3,4) $$ where $\langle i\rangle=\{\pm I,\pm iI\}\simeq\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.

More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows: $$ a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$

It is routine to check that $\langle i\rangle\simeq\mathbb{Z}_4$ and the three generators $a,b,c$ of $D(2,3,4)\simeq S_4$ preserve the matrices $\mathcal{P}$.

Therefore, the answer of this question is the same $K$ as above.