I am working on a particular problem: I generate the set of $n\times n$ permutation matrices. I initialise my problem $$K=PXP^{\dagger}$$ where $P$ is some permutation matrix and $X$ is the usual Pauli-X/Sigma-X matrix. Then I iterate over this definition, setting $P$ to $K$, and repeating. I have $n!$ permutation matrices, and as far as I understand this forms $n!/n$ equivalence classes of length $n$. For example for $n=3$, I have the two sets; $$\{\mathbb{I},X,X^2\} \\ \{P,XP,X^2P\}$$ where $$P=\begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}$$ So it is not necessary to iterate over all 6 permutation matrices, instead I can just iterate over $\mathbb{I}$ and $P$. In general, I have found it is only necessary to iterate over the $n\times n$ permutation matrices where the top left entry is 1.
I'm struggling to write this as an equivalence relation, or somehow compact this into a sentence. It may be worth saying I am a undergraduate physicist who has not studied much group theory. I am not asking for the exact answer, but any guidance would be great.
Edit - The $n\times n$ Pauli-X matrix is defined as $$X_n=\begin{bmatrix} \ 0 & 0 & 0 & \dots & 0 & 1\ \ \\ \ 1 & 0 & 0 & \dots & 0 & 0\ \ \\ \ 0 & 1 & 0 & \dots & 0 & 0\ \ \\ \ 0 & 0 & 1 & \dots & 0 & 0\ \ \\ \ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \ \ \\ \ 0 & 0 & 0 & \dots & 1 & 0\ \ \\ \end{bmatrix}$$