Given a square matrix $A=[a_{ij}]_{n \times n}$, an operation $swap(A, i, j)$ is defined to swap row $i$ and $j$ of $A$ and do the same thing with the corresponding columns. For example, in the following example, we can see the $swap(A, 1, 3)$:
$$A=\left[ \begin{array}{ccc} a & b & c \\ d & e & f \\ g & h & i \end{array} \right] ~~~~~\Rightarrow~~~~~ swap(A,1,3)=\left[ \begin{array}{ccc} i & h & g \\ f & e & d \\ c & b & a \end{array} \right]$$
For a matrix $A$, let $swap^k(A)$ be the set of matrices produced by exactly $k$ swaps. For example $B \in swap^3(A)$ means that there is a sequence of 3 swaps that can convert $A$ into $B$.
A unique encoding $\phi$ of this matrix is a function of $A$ such that:
$$\phi(A)=\phi(B) ~~~~ \Leftrightarrow ~~~~ \exists k ~:~ B \in swap^k(A)$$
For simplicity, we can assume that the matrix is symetric and its elements are in $\{0,1\}$. I am interested to know whether there exist such encoding for this case of matrices or if there is a proof for its non-existence.
Edit. What is the most efficient way to do this?
Such an encoding exists and is finite. Whether or not it is tractable is a much harder question.
$\phi(A)$ can be defined to be an encoding of the span of the swaps of $A$:
$$\text{SwapSpan}(A) = \cup_{k=0}^{\infty} \text{swap}^k(A)$$ $$\phi(A) = \text{Encoding}(\text{SwapSpan}(A))$$
Since there are $n^2$ elements of the matrix, there is an upper bound of $(n^2)!$ number of matrices in $\text{SwapSpan}(A)$. Explicitly writing out the list of matrices in lexicographical order is a valid canonical $\text{Encoding}$.
Note that if $A \in \text{SwapSpan}(B)$ then $\text{SwapSpan}(A) = \text{SwapSpan}(B)$ since the order of the swaps can simply be reversed.
The span can also be computed, since a breadth first search of all the swaps is guaranteed to terminate.
Whether or not this is tractable depends on your computing power and the size of your matrices.