Let $O(n)$ be the set of $3 \times 3$ orthogonal matrices, and $S$ be the set of traceless real symmetric $3 \times 3$ matrices.
Now $O(n)$ will act on $S$ by conjugation. Note that by the spectral theorem $ \forall A \in S$ in the orbit of $A$ there is a diagonal matrix $D=diag(\lambda_1, \lambda_2,\lambda_3)$ with $\lambda_1\le\lambda_2 \le\lambda_3 $.
Determine the all the orbits of the action & the orbit space of the action.
What I've done:
Since all elements of $S$ are diagonlizable that the orbit of any element should be the orbit of its' diagonalized form because $\forall A \in S, \exists P \in O(3)$ st $PAP^t=D $ with $D$ a diagonal matrix and moreover for any other $Q\in O(3)$ we get $QDQ^t=QPAP^tQ^t=(QP)A(QP)^t=RAR^t$.
And so would that imply the orbit space is just the set of traceless diagonal matrices as $tr(QDQ^t)=tr(DQ^tQ)=tr(Q)$ (cyclic property of trace) and $Q\in S$.
Any help is appreciated.
Not exactly. Note that if you permute the entries of a diagonal matrix along the diagonal, the resulting diagonal matrix belongs to the same orbit. So, the orbit space is smaller than what you've suggested.