Subgroups and invariants in a unitary group U(3)

Question

This is related to the post, but an enriched version of the problem. Now we require the richer form of $P_1,P_2,P_3,P_4,P_5,P_6$.

Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices.

Can we find some subgroup of Lie group, $$k \in K \subset G= U(3) $$ such that

$$ k^T \{P_1, P_2, P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6 \} k =\{P_1, P_2, P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}. $$ This means that set $\{P_1, P_2, P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}$ is invariant under the transformation by $k$. Here $k^T$ is the transpose of $k$. What is the full subset (or subgroup) of $K$?

Here we define: $$ P_1 = \left( \begin{array}{ccc} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_2 = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_3 = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{array} \right).$$ $$ P_4 =\sqrt{2} \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_5 = \sqrt{2}\left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_6 = \sqrt{2}\left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \\ \end{array} \right).$$

This means that $k^T P_a k= \pm P_b$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2,3,4,5,6 \}$. But overall the full set $ \{P_1, P_2, P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}$ is invariant under the transformation by $k$.

There must be a trivial element $k=$ the rank-3 identity matrix. But what else can it allow?

How could we determine the complete $K$?

Answer 1

The answer (and the method) is the same as the previous question.

The subgroup $K$ of $U(3)$ containing invariant matrices are isomorphic to the finite group $$ \mathbb{Z}_4\times S_4 \cong\langle i\rangle\times D(2,3,4) $$ where $\langle i\rangle=\{\pm I,\pm iI\}\cong\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.

More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows: $$ a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$

1. It is a fact from elementary linear algebra that $P_a$ and $k^TP_ak$ have the same rank because $k\in U(3)$ is non-singular. Notice that $P_1,P_2,P_3$ are of rank $2$, but $P_4,P_5,P_6$ are of rank $1$. Thus, if $k^TP_ak=\pm P_b$, then either $a,b\in\{1,2,3\}$ or $a,b\in\{4,5,6\}$.

2. All the matrices preserving $\{P_i\mid 1\leq i\leq 6\}$ also preserve $\{P_i\mid 1\leq i\leq 3\}$.

3. Because we already know all the matrices preserving $\{P_i\mid 1\leq i\leq 3\}$ in the previous question, it suffices to check whether those matrices preserve $\{P_i\mid 4\leq i\leq 6\}$ or not.

4. Note that the three generators $a,b,c$ preserve $\{P_i\mid 4\leq i\leq 6\}$ as follows $$ \begin{gather*} a^TP_4a=P_4, \quad b^TP_4b=P_5, \quad c^TP_4c=P_5 \\ a^TP_5a=P_6, \quad b^TP_5b=P_6, \quad c^TP_5c=P_4 \\ a^TP_6a=P_5, \quad b^TP_6b=P_4, \quad c^TP_6c=P_6 \end{gather*} $$

5. It is trivial that $\langle i\rangle=\{\pm I,\pm iI\}$ preserve $\{P_i\mid 4\leq i\leq 6\}$.

6. Therefore we have the same solution as the previous question.