Invariant subgroup in a self-conjugate pair multiplication in Unitary Group $U(3)$

Question

Inspired by @ChoF and @annie heart's exploration, I like to post a similar but possibly richer structure related to the decompositions of $U(3)$ or $SU(3)$ representation multiplications $$3 \otimes \bar 3 = 1 \oplus 8.$$ 3 is in the fundamental and 8 is in the adjoint representation of $SU(3)$. Below I modify and preserve the structure of questions from (Subgroups and invariants in a unitary group U(3))

Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices. Now we take a set of $P$ matrices from Gell-Mann matrices.

Can we find some subgroup of Lie group, $$k \in K \subset G= U(3) $$ such that

$$ k^\dagger \{\pm P_1, \pm P_2,\pm P_3,\pm P_4,\pm P_5, \pm P_6, \pm P_7, \pm P_8 \} k$$ $$=\{\pm P_1, \pm P_2,\pm P_3,\pm P_4,\pm P_5, \pm P_6, \pm P_7, \pm P_8\}. $$ This means that the full set $\{\pm P_1, \pm P_2,\pm P_3,\pm P_4,\pm P_5, \pm P_6, \pm P_7, \pm P_8\}$ is invariant under the transformation by $k$. Here $$k^\dagger \equiv (k^*)^T$$ is the complex conjugate ($*$) transpose ($T$) of $k$. What is the full subset (or subgroup) of $K \subset G$?

Here we define: $$ P_1 = \left( \begin{array}{ccc} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_2 = \left( \begin{array}{ccc} 0 & -i & 0 \\ i & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_3 = \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 0 \\ \end{array} \right) $$ $$ P_4 = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 1 & 0 & 0 \\ \end{array} \right),\;\;\;\; P_5 = \left( \begin{array}{ccc} 0 & 0 & -i \\ 0 & 0 & 0 \\ i & 0 & 0 \\ \end{array} \right),\;\;\;\;$$ $$P_6 = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ \end{array} \right),\;\;\;\; P_7 = \left( \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & -i \\ 0 & i & 0 \\ \end{array} \right).$$ $$P_8 =\frac{1}{\sqrt 3} \left( \begin{array}{ccc} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -2 \\ \end{array} \right).$$

This means that $$k^\dagger P_a k= \pm P_b$$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2,3,4,5,6,7,8 \}$. But overall the full set $ \{\pm P_1, \pm P_2,\pm P_3,\pm P_4,\pm P_5, \pm P_6, \pm P_7, \pm P_8 \}$ is invariant under the transformation by $k$.

Please determine the complete $K$.

p.s. I am not so sure the $P_8$ is in the most symmetric form in accordance with $P_j$ $j \in \{1,2,3,4,5,6,7 \}$. So maybe one can suggest the possible modification of $P_8$ to have a larger $K \subset U(3)$

Answer 1

Note that the transformation $k^\dagger P_a k$ preserves the eigenvalue information of $P_a$ so that it preserves trace, determinant, and rank, etc.

1. Since only $P_8$ has different eigenvalues, we must have $k^\dagger P_8k=P_8$. Furthermore, any $k\in U(3)$ satisfying $k^\dagger P_8k=P_8$ has the form $\begin{pmatrix} U(2) & 0 \\ 0 & U(1) \end{pmatrix}$.

2. Any $k\in U(2)\times U(1)$ transforms either (1st case) $$ \{\pm P_4,\pm P_5\}\leftrightarrow\{\pm P_4,\pm P_5\} \quad\text{and}\quad \{\pm P_6,\pm P_7\}\leftrightarrow\{\pm P_6,\pm P_7\} $$ or (2nd case) $\{\pm P_4,\pm P_5\}\leftrightarrow\{\pm P_6,\pm P_7\}$.

3. In the (1st case), $k$ has the form $\begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix}\in U(1)\times U(1)\times U(1)$, and

   in the (2nd case), $k$ has the form $\begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix}\in U(1)\times U(1)\times U(1)$,

   where $\alpha,\beta,\gamma\in U(1)$ satisfy $$ (\alpha=\pm\gamma \text{ or} \pm i\gamma) \quad\text{and}\quad (\beta=\pm\gamma \text{ or} \pm i\gamma) $$

4. Moreover, in both cases, $k$ transforms $\{\pm P_1,\pm P_2\}\leftrightarrow\{\pm P_1,\pm P_2\}$ and $\{\pm P_3\}\leftrightarrow\{\pm P_3\}$.

Answer. The invariant subgroup $K$ of $U(3)$ is isomorphic to the infinite group $$ K \simeq U(1) \times G $$ where $G = (\mathbb{Z}_4\times\mathbb{Z}_4) \rtimes_\varphi \mathbb{Z}_2$ (GAP ID [32,11]).

More specifically, for $\gamma\in U(1)$, the elements in $K$ are $$ \gamma \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & 1 \end{pmatrix} \quad\text{or}\quad \gamma \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix} $$ where $\alpha,\beta\in\{\pm1,\pm i\}$.