What sets of orthogonal matrices $A$ and $B$ satisfy $AABBAB=BBAABA$?

Question

Given two continuous curves of orthogonal matrices $t\to A_t$ and $ t\to B_t$, such that for any value of $t$ we have:

• $AABBAB=BBAABA$
• $A$ and $B$ do not commute.

What solutions are there for $A_t$ and $B_t$?

Edit: Without loss of generality, let $t \in (0,1)$.

Answer 1

The GAP code

gap> F := FreeGroup(2);                                  
<free group on the generators [ f1, f2 ]>
gap> a := F.1;; b := F.2;;                                                
gap> G := F / [a*a*b*b*a*b*(b*b*a*a*b*a)^(-1), a^4, b^3];
<fp group on the generators [ f1, f2 ]>
gap> StructureDescription(G);                            
"(C5 x C5) : C12"

shows that there are matrices $A$ and $B$ which do not commute, satisfy your relations, and moreover have $A^4=B^3=I$. Looking at the character table of the group shows that it has faithful representations of degree $12$, so one can get $n\times n$ orthogonal matrices satisfying those relations for all $n\geq12$.