Expression of unitary group , the discrete subgroups and invariants

Question

Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices.

1. What is the convenient way to parametrize the rank-3 matrix in terms of a 9 degrees of freedom (for 9 generators)?

2. Can we find some subgroup of Lie group, $$k \in K \subset G= U(3) $$ such that

   $$ k^T \{R_1, R_2\} k =\{R_1, R_2\} . $$ This means that set $\{R_1, R_2\}$ is invariant under the transformation by $k$. Namely, both cases are allowed: $$ k^T R_1 k =R_1,\;\;\; k^T R_2 k =R_2 . $$ $$ k^T R_1 k =R_2,\;\;\; k^T R_1 k =R_2 . $$

Here $k^T$ is the transpose of $k$. What is the full subset (or subgroup) of $K$?

Here we define: $$ R_1 = \left( \begin{array}{ccc} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right),\;\;\;\; R_2 =-R_1= -\left( \begin{array}{ccc} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 0 \\ \end{array} \right).$$

This means that $k^T R_a k=R_b$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2 \}$. But overall the full set $ \{R_1, R_2\}$ is invariant under the transformation by $k$.

There must be a trivial element $k=$ the rank-3 identity matrix. But what else can it allow? In particular, I can see an SU(2) and an additional $\mathbb{Z}_2 \times \mathbb{Z}_2$ structure in $K$.

How could we determine the complete $K$?

Edit: More clarifications. Simplified the problem.

Answer 1

Question. Let $R_1=\begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}$. Find the subgroup $K$ of $U(3)$, where $$ K=\{k\in U(3)\mid k^TR_1k=R_1\text{ or }-R_1\} $$

Answer. Denote by $SU(2)\rtimes\mathbb{Z}_2$ the group of $2\times2$ unitary matrices with determinant $\pm1$. Then $$ \begin{align*} K &= \Bigl\{ \begin{pmatrix} \alpha & \mp\bar\beta & 0 \\ \beta & \pm\bar\alpha & 0 \\ 0 & 0 & \gamma \end{pmatrix} \in U(3) \mid \alpha,\beta,\gamma\in\mathbb{C},\,|\alpha|^2+|\beta|^2=|\gamma|=1 \Bigr\} \\ &\cong (SU(2)\rtimes\mathbb{Z}_2)\times U(1) \end{align*} $$

Solution. Let us find $k=(k_{ij})$ such that $k^TR_1k=R_1$ or $-R_1$. It implies the following three equations $$ k_{11}k_{23} = k_{13}k_{21}, \quad k_{12}k_{23} = k_{13}k_{22}, \quad k_{11}k_{22} - k_{12}k_{21} = \pm1 \tag{*} $$

Claim 1. $k_{13}=0$ and $k_{23}=0$.

Proof. If $k_{13}\neq 0$, then by the Gaussian elimination $$ k=\begin{pmatrix} k_{11} & k_{12} & k_{13} \\ k_{21} & k_{22} & k_{23} \\ k_{31} & k_{32} & k_{33} \end{pmatrix} \sim \begin{pmatrix} k_{11} & k_{12} & k_{13} \\ k_{21}-k_{11}(\frac{k_{23}}{k_{13}}) & k_{22}-k_{12}(\frac{k_{23}}{k_{13}}) & k_{23}-k_{13}(\frac{k_{23}}{k_{13}}) \\ k_{31} & k_{32} & k_{33} \end{pmatrix} \sim \begin{pmatrix} k_{11} & k_{12} & k_{13} \\ 0 & 0 & 0 \\ k_{31} & k_{32} & k_{33} \end{pmatrix} $$ $k$ is singular which contradicts to $k\in U(3)$. In the same way, we can prove that $k_{23}=0$ too.

Now the three equations in (*) are reduced to only one equation $$ k_{11}k_{22} - k_{12}k_{21} = \begin{cases} +1 & \text{if $k^TR_1k=R_1$} \\ -1 & \text{if $k^TR_1k=-R_1$} \end{cases} \tag{**} $$

Claim 2. $k_{31}=k_{32}=0$.

Proof. Since $k\in U(3)$, $$ k^\dagger k=\begin{pmatrix} * & * & \bar k_{31} \\ * & * & \bar k_{32} \\ 0 & 0 & \bar k_{33} \end{pmatrix} \begin{pmatrix} * & * & 0 \\ * & * & 0 \\ k_{31} & k_{32} & k_{33} \end{pmatrix} = \begin{pmatrix} * & * & * \\ * & * & * \\ k_{31}\bar k_{33} & k_{32}\bar k_{33} & k_{33}\bar k_{33} \end{pmatrix} = I $$ where $k^\dagger$ denotes the conjugate transpose of $k$. From $k_{33}\bar k_{33}=|k_{33}|^2=1$, we have $\bar k_{33}\neq0$ so that $k_{31}=k_{32}=0$.

Now we have $k=\begin{pmatrix} k_{11} & k_{12} & 0 \\ k_{21} & k_{22} & 0 \\ 0 & 0 & k_{33} \end{pmatrix}\in U(2)\times U(1)$ where $|k_{33}|=1$, and the equation (**) implies that the determinant of the sub-matrix $\begin{pmatrix} k_{11} & k_{12} \\ k_{21} & k_{22} \end{pmatrix}\in U(2)$ is $\pm1$.

On the contrary, it is easy to check this form of $k$ satisfies $k^TR_1k=R_1$ or $-R_1$.

Note. For the first question, I do not know any convenient way to parametrize $U(3)$.