Orthogonal transformation with additional constraints

Question

Let $A$ be an orthogonal matrix, i.e. $AA^{T}=\mathbb{I}$. It is given that $A$ satisfies an additional constraint, $AMA^{T}=PMP^{T}$, where $P$ is some permutation matrix and $M_{ij}=sgn(i-j)$. Can $A$ be still an arbitrary orthogonal matrix or are there any new constraints on it? In two dimension there are no new restrictions.

Answer 1

No. The matrix $PMP^T$ has integer entries. That's not usually the case with $AMA^T$. For instance, with $$ A=\begin{bmatrix} \tfrac1{\sqrt{14}}&\tfrac3{\sqrt{14}}&\tfrac2{\sqrt{14}}\\ \tfrac5{\sqrt{42}}&\tfrac1{\sqrt{42}}&-\tfrac4{\sqrt{42}}\\ \tfrac1{\sqrt3}&-\tfrac1{\sqrt3}&\tfrac1{\sqrt3} \end{bmatrix}, $$ The 1,2 entry of $AMA^T$ is $\sqrt3$.

Answer 2

Your claim is wrong even when $n=2$. E.g. when $P=I_2$, we need $AM=MA$. Since $M$ in this case is precisely the rotation matrix $R$ for angle $\pi/2$, it only commutes with scalar multiples of rotation matrices. Therefore $A$ must have determinant $1$. Similarly, if $P$ is a transposition matrix, $A$ must have determinant $-1$.

In general, since $M$ is a skew symmetric matrix with distinct eigenvalues (namely, $\tan\left(\frac{k\pi}{2n}\right)i$ for $k=1-n,\,3-n,\ldots,\,n-3,\,n-1$), it is orthogonally similar to a direct sum of different nonzero scalar multiples of $R$ (and also the scalar $0$ if $n$ is odd). It follows that $P^TA$, via the same similarity transform, must be equal to a direct sum of $2\times2$ rotation matrices (and also a scalar $\pm1$ when $n$ is odd).