I am asked to compute the orbits and isotropy groups (a.k.a. stabilizers) of the Lie group action \begin{align*} \varphi\colon \operatorname{SO}(3)\times\operatorname{Sym}(3) & \rightarrow \operatorname{Sym}(3) \\ (R,A) &\mapsto R A R^t \end{align*} where $\operatorname{Sym}(3)$ is the space of symmetric $3\times 3$ matrices. If I did it well, for an element $A\in\operatorname{Sym}(3)$, its orbit $\mathcal{O}(A)$ and isotropy group $G_A$ are given by:
\begin{equation*} \mathcal{O}(A) = \{B\in\operatorname{Sym}(3) \ \big| \ \operatorname{Eigenvalues}(B)=\operatorname{Eigenvalues}(A) \} \end{equation*}
\begin{align*} G_A \cong \begin{cases} \operatorname{SO}(3) &\text{ if $A$ has 3 identical eigenvalues}\\ \operatorname{O}(2) &\text{ if $A$ has 2 identical eigenvalues}\\ F &\text{ if $A$ has no identical eigenvalue} \end{cases} \end{align*} where $F$ is the finite subgroup $\{I,\ \operatorname{diag}(1,-1,-1),\ \operatorname{diag}(-1,1,-1),\ \operatorname{diag}(-1,-1,1)\}$ of $\operatorname{SO}(3)$.
What I did is mostly to use standard linear algebra results for the case of $A$ diagonal and then extend to the non-diagonal case using that:
- Any $A$ is always on the same orbit of a diagonal matrix with the same eigenvalues.
- For two elements on the same orbit, their isotropy groups are conjugated and, therefore, isomorphic.
Now I am curious about if there is a more "sophisticated" way of computing these orbits and stabilizers using some results or techniques about representation theory. I don't know much about it, but I believe that the following map is a representation of $\operatorname{SO}(3)$: \begin{align*} \Pi \colon \operatorname{SO}(3) &\rightarrow \operatorname{GL}(\operatorname{Sym}(3)) \\ R &\mapsto \varphi_R \end{align*} where $\varphi_R(A) = \varphi(R,A) = R A R^t$.
Thanks a lot for your help!