For orthogonal matrices $A$ and $B$, does $ABAB=BABA$ imply $A$ and $B$ commute?

Question

Given two distinct orthogonal matrices $A$ and $B$, given some individual sequence of applications of these matrices such that each matrix appears an equal number of times (e.g. $AABBAB$, with $3$ occurrences of each matrix), if the application of this sequence is equal to the application of the sequences dual, where each of our matrices is replaced by the other (e.g. $AABBAB=BBAABA$), do $A$ and $B$ necessarily commute ($AB=BA$)?

This is clearly a sufficient condition, but is it a necessary one?

I am trying to show $A$ and $B$ must belong to a subgroup of $O(n)$ isomorphic to $O(2)$.

It also bears note that both $A$ and $B$ must vary continuously according to some parameter $t\in (0,1)$ while retaining this property.

Edit: it seems that should $A$ and $B$ satisfy this condition, then $A$ may be replaced by any element $B^ZAB^{-Z}$.

To clarify my question: if one sequence of applications of the two matrices $A$ and $B$ in equal number is equal to the dual of that sequence made by swapping the matrices, are the matrices $A$ and $B$ necessarily commutative?

Answer 1

It is easy to construct two matrices $A$ and $A$ such that $(AB)^n=(BA)^n$ for some $n$. Take two reflections with respect to lines that form an angle of $2\pi/n$. They don't commute.

On a different line, take two matrices $A$ and $B$ of order $3$ that do not commute and notice that $AAABBB=BBBAAA$.

Answer 2

To answer the question in the title: no.

The condition $ABAB = BABA$ can be equivalently written as $(AB)^2 = (BA)^2$. Can we have orthogonal matrices $A$ and $B$ such that $(AB)^2 = (BA)^2$ without having $AB = BA$?

The answer is yes, because this can happen in finite groups! For example, in the quaternion group $Q_8$, we have $(ij)^2 = k^2 = -1 = (-k)^2 = (ji)^2$, but $ij = k \neq -k = ji$.

Every finite group admits a faithful orthogonal action, so this answers the question in the negative. This example in $Q_8$ yields the matrices

$$A = \left( \begin{array}{cccc} 0 & -1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & -1 \\ 0 & 0 & 1 & 0 \\ \end{array} \right), \quad B = \left( \begin{array}{cccc} 0 & 0 & -1 & 0 \\ 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 0 \\ 0 & -1 & 0 & 0 \\ \end{array} \right)$$

which are indeed orthogonal and satify $ABAB=BABA$, but do not commute.

Answer 3

Let $$A = \left(\begin{array}{cc} 1 & 0 \\ 0 & -1 \end{array}\right), \quad B = \left(\begin{array}{cc} 0 & 1 \\ 1 & 0 \end{array}\right).$$ Then $$BA = \left(\begin{array}{cc} 0 & -1 \\ 1 & 0 \end{array}\right) \ne \left(\begin{array}{cc} 0 & 1 \\ -1 & 0 \end{array}\right) = AB.$$ But $(AB)^2=(BA)^2=-I_2$.