I'm interested in the invariants of a symmetric matrix under the group of permutation matrices. This is a subgroup of the group of orthogonal matrices, for which there are results on Google, but I'm interested in this particular subgroup.
To be more precise, I'm looking for invariants for the action $PAP^t$ where $A$ is a symmetric matrix and $P$ is a permutation matrix. The case where $P$ is orthogonal is known.
Two points that you might find interesting.
First, any symmetric matrix can be interpreted as the adjecency matrix of a graph (undirected), potentially with self-loops. Two symmetric matrices will be permutation similar (i.e. will correspond to the same orbit) if and only if their corresponding graphs are isomorphic. Thus, any invariant can be interpreted as a property of this weighted graph, and conversely any property of this weighted graph defines an invariant.
Second, it is notable that for a matrix whose diagonal entries are distinct, there is a unique element of its equivalence class whose diagonal entries are in increasing order.