Discrete non-Abelian subgroups and invariants in a unitary group

Question

Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices.

1. Can we find some subgroup of Lie group, $$k \in K \subset G= U(3) $$ such that

   $$ k^T \{P_1, P_2, P_3, -P_1, - P_2, - P_3 \} k =\{P_1, P_2, P_3, -P_1, - P_2, - P_3\} . $$ This means that set $\{P_1, P_2, P_3, -P_1, - P_2, - P_3\}$ is invariant under the transformation by $k$. Here $k^T$ is the transpose of $k$. What is the full subset (or subgroup) of $K$?

Here we define: $$ P_1 = \left( \begin{array}{ccc} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_2 = \left( \begin{array}{ccc} 0 & 0 & -1 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_3 = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & -1 & 0 \\ \end{array} \right).$$

This means that $k^T P_a k= \pm P_b$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2,3 \}$. But overall the full set $ \{P_1, P_2, P_3, -P_1, - P_2, - P_3\}$ is invariant under the transformation by $k$.

There must be a trivial element $k=$ the rank-3 identity matrix. But what else can it allow? In particular, I can see a symmetric group S$_4$ and $\mathbb{Z}_3$ and a possible additional structure in $K$.

How could we determine the complete $K$?

Answer 1

In the OP's previous question I already proposed an answer as follows:

The subgroup $K=\{k\in U(3)\mid k^TP_1k=\pm P_1\}$ of $U(3)$ is isomorphic to $(SU(2)\rtimes\mathbb{Z}_2)\times U(1)$ where $SU(2)\rtimes\mathbb{Z}_2$ is the group of $2\times2$ unitary matrices with determinant $\pm1$.

In particular, we have a specific matrix form $$ \begin{align*} K &= \Bigl\{ \begin{pmatrix} \alpha & \mp\bar\beta & 0 \\ \beta & \pm\bar\alpha & 0 \\ 0 & 0 & \gamma \end{pmatrix} \in U(3) \mid \alpha,\beta,\gamma\in\mathbb{C},\,|\alpha|^2+|\beta|^2=|\gamma|=1 \Bigr\} \\ &= \Bigl\{ \begin{pmatrix} A & \mathbf{0} \\ \mathbf{0} & \gamma \end{pmatrix} \in U(3) \mid A\in U(2),\,\det A=\pm1,\,|\gamma|=1 \Bigr\} \tag{$*$} \end{align*} $$

At first we fix a permutation matrix $Q=\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}$.

Note that the order of $Q$ is $3$, and $Q^2=Q^{-1}=Q^T$. Moreover we have $$ QP_1Q^T=P_2 \quad\text{and}\quad Q^2P_1(Q^2)^T=Q^TP_1Q=P_3 $$

Claim. The set of matrices $\{\bar k\in U(3) \mid \bar k^TP_1 \bar k=\pm P_2\}$ is $KQ^T=\{kQ^T\mid k\in K\}$.

Proof. If $k\in U(3)$ satisfies $k^TP_1k=\pm P_1$, then $k\in K$ (see the equation ($*$) above), and $$ (kQ^T)^TP_1(kQ^T)=Q(k^TP_1k)Q^T=\pm(QP_1Q^T)=\pm P_2 $$ On the contrary, if $\bar k\in U(3)$ satisfies $\bar k^TP_1\bar k=\pm P_2$, then $$ \bar k^TP_1\bar k=\pm P_2=\pm(QP_1Q^T) \implies (\bar kQ)^TP_1(\bar kQ)=\pm P_1 $$ so that $\bar kQ=k \Rightarrow \bar k=kQ^T$ for some $k\in K$.

In a similar way, it is routine to check that $$ \begin{align*} K_{00} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_1\} &= Q^0KQ^0 = K \\ K_{02} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_2\} &= Q^0KQ^2 = KQ^T \\ K_{01} \equiv \{k\in U(3) \mid k^TP_1 k=\pm P_3\} &= Q^0KQ^1 = KQ \\ K_{10} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_1\} &= Q^1KQ^0 = QK \\ K_{12} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_2\} &= Q^1KQ^2 = QKQ^T \\ K_{11} \equiv \{k\in U(3) \mid k^TP_2 k=\pm P_3\} &= Q^1KQ^1 = QKQ \\ K_{20} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_1\} &= Q^2KQ^0 = Q^TK \\ K_{22} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_2\} &= Q^2KQ^2 = Q^TKQ^T \\ K_{21} \equiv \{k\in U(3) \mid k^TP_3 k=\pm P_3\} &= Q^2KQ^1 = Q^TKQ \end{align*} $$

Now let us find $\{\pm P_1,\pm P_2,\pm P_3\}$ invariant matrices. There are three cases $K_{00},K_{02},K_{01}$ according to $k^TP_1k=\pm P_1$, $\pm P_2$ or $\pm P_3$.

Case 1. Find the intersection of $K_{00}$ with $K_{10},K_{12},K_{11}$ according to the type of $k^TP_2k$. $$ \begin{align*} K_{00}\cap K_{10} &= K \cap QK = \varnothing \\ K_{00}\cap K_{12} &= K \cap QKQ^T = \bigl\{ \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix} \text{ where $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$} \bigr\} \\ K_{00}\cap K_{11} &= K \cap QKQ = \bigl\{ \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix} \text{ where $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$} \bigr\} \end{align*} $$

Now find the intersection with the remaining case $K_{20},K_{22},K_{21}$. $$ \begin{align*} K_{00}\cap K_{12}\cap K_{20} &= K \cap QKQ^T \cap Q^TK= \varnothing \\ K_{00}\cap K_{12}\cap K_{22} &= K \cap QKQ^T \cap Q^TKQ^T = \varnothing \\ K_{00}\cap K_{12}\cap K_{21} &= K \cap QKQ^T \cap Q^TKQ \\ &= \bigl\{ \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix} \text{ where $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$, $\beta\gamma=\pm1$} \bigr\} \\ K_{00}\cap K_{11}\cap K_{20} &= K \cap QKQ \cap Q^TK = \varnothing \\ K_{00}\cap K_{11}\cap K_{22} &= K \cap QKQ \cap Q^TKQ^T \\ &= \bigl\{ \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix} \text{ where $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$, $\beta\gamma=\pm1$} \bigr\} \\ K_{00}\cap K_{11}\cap K_{21} &= K \cap QKQ \cap Q^TKQ = \varnothing \end{align*} $$

We can find intersections for the other cases $K_{02}$ and $K_{01}$ in a similar way.

Finally, we have the following $16\times 6$ invariant matrices:

For $\alpha,\beta,\gamma\in\mathbb{C}$ satisfying $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$, and $\beta\gamma=\pm1$, $$ \begin{align*} K_{00}\cap K_{12}\cap K_{21} &\Rightarrow \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix} \quad K_{00}\cap K_{11}\cap K_{22} \Rightarrow \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix} \\ K_{02}\cap K_{10}\cap K_{21} &\Rightarrow\begin{pmatrix} \alpha & 0 & 0 \\ 0 & 0 & \beta \\ 0 & \gamma & 0 \end{pmatrix} \quad K_{02}\cap K_{11}\cap K_{20} \Rightarrow\begin{pmatrix} 0 & 0 & \alpha \\ \beta & 0 & 0 \\ 0 & \gamma & 0 \end{pmatrix} \\ K_{01}\cap K_{10}\cap K_{22} &\Rightarrow\begin{pmatrix} 0 & \alpha & 0 \\ 0 & 0 & \beta \\ \gamma & 0 & 0 \end{pmatrix} \quad K_{01}\cap K_{12}\cap K_{20} \Rightarrow\begin{pmatrix} 0 & 0 & \alpha \\ 0 & \beta & 0 \\ \gamma & 0 & 0 \end{pmatrix} \end{align*} $$

Note that there are 16 solutions satisfying $\alpha\beta=\pm1$, $\alpha\gamma=\pm1$, $\beta\gamma=\pm1$, which are $\alpha,\beta,\gamma=\pm1$ (8 solutions of order 1 or 2) and $\alpha,\beta,\gamma=\pm i$ (8 solutions or order 4).

Therefore the subgroup of $U(3)$ containing invariant matrices are isomorphic to the finite group $$ (\mathbb{Z}_2\times\mathbb{Z}_2\times\mathbb{Z}_4) \rtimes S_3 \cong \mathbb{Z}_4\times S_4 \cong\langle i\rangle\times D(2,3,4) $$ where $\langle i\rangle=\{\pm I,\pm iI\}\cong\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.

More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows: $$ a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$

(The idea of this isomorphism was suggested by @wonderich.)