I can define one version of a fractional-transpose quite easily by interpolation between the exchanged elements:
- Take $\mathbf{U}(\mathbf{A})$ as the upper triangle of $\mathbf{A}$ excluding the main diagonal of $\mathbf{A}$.
- Take $\mathbf{L}(\mathbf{A})$ as the lower triangle of $\mathbf{A}$ excluding the main diagonal of $\mathbf{A}$.
- Let $\mathbf{D}$ be the matrix with the same diagonal of $\mathbf{A}$ and zeros elsewhere.
Then I can form: $$T_\alpha(\mathbf{A}) = \mathbf{D}+\left[\alpha\mathbf{U}(\mathbf{A}^\top)+(1-\alpha)\mathbf{U}(\mathbf{A})\right]+\left[\alpha\mathbf{L}(\mathbf{A}^\top)+(1-\alpha)\mathbf{L}(\mathbf{A})\right]$$
Then we have $T_0(\mathbf{A})=\mathbf{A}$ and $T_1(\mathbf{A})=\mathbf{A}^\top$.
However, this doesn't have the nice properties that $T_{1-\alpha}(T_{\alpha}(\mathbf{A}))=\mathbf{A}^\top$ and $T_{1-\alpha}(T_{\alpha}(\mathbf{A}^\top))=\mathbf{A}$.
Can you help me come up with an operation $T_\alpha$ that can be repeatedly applied like this to get the transpose?
Let's consider the space of $2\times2$ matrices. As noted here, the transposition operator acting on the elements of such space can be represented by a $4\times4$ matrix. In particular, if we take the $2\times2$ matrix
$$A=\begin{pmatrix} a&b\\ c&d \end{pmatrix},$$
then, after flattening it into a column vector
$$A'=\begin{pmatrix}a\\ b\\ c\\ d\end{pmatrix},$$
we can represent the transposition operator with the following permutation matrix:
$$T_1=\begin{pmatrix} 1&0&0&0\\ 0&0&1&0\\ 0&1&0&0\\ 0&0&0&1 \end{pmatrix}.$$
If you now take square root of this matrix, you'll get the half-transposition operator representation:
$$T_{1/2}=\begin{pmatrix} 1&0&0&0\\ 0& \frac12+\frac i2 & \frac12-\frac i2 &0\\ 0& \frac12-\frac i2 & \frac12+\frac i2 &0\\ 0&0&0&1 \end{pmatrix}.$$
Similarly you can take other powers of $T_1$ to get the general matrix $T_\alpha$. Also similarly, you can extend this to higher-dimensional $A$-matrices.