Recently when teaching the basics of matrices, a student asked if there are matrices $A,B$ for which $AB\ne BA$, but for some $n>1$ we have $(AB)^n = (BA)^n$? I was able to come up with a few cases where the answer is yes.
- If both $AB$ and $BA$ are nilpotent, then the result is obvious; take $n$ to be the maximum power that kills both.
- Another nice case is if $A,B$ are rotation matrices in dimension $d\ge 3$ (so they don't commute) and are both rotations by rational multiples of $2\pi$. In particular, the case where $d=3$ and $A$ is a rotation about the $x$-axis by $\pi/2$ and $B$ is a rotation about the $y$-axis by $\pi/2$ readily gives $AB\ne BA$ but $(AB)^3=(BA)^3=I$. The student is a fan of Rubik's Cubes so I think this will make a nice example.
- My gut says other geometric-flavored matrices (i.e., reflections) will also have this property.
However, I want to ask a follow-up question, namely to classify all such matrices with this property. Writing $A=PS P^{-1}$ and $B=PT P^{-1}$ for some matrix $P$ is intractable, as we have $$ (AB)^n = P (ST)^n P^{-1} ; (BA)^n = P(TS)^n P^{-1}, $$which gets us nowhere. Is there some nice classification of these matrices?
Not a complete answer, but more than an extended comment.
If $A$ is invertible, then $AB$ and $BA$ are conjugate. We have $(AB)^n=(A(BA)A^{-1})^n=A(BA)^nA^{-1}$, so $(AB)^n=(BA)^n$ if and only if $A$ commutes with $(BA)^n$. Hence, we are looking for $A$ such that $A$ does not commute with $BA$ but does commute with $(BA)^n$.
Using this, if $A$ is invertible and $BA$ is diagonalizable, then one can show that we can change basis such that $A$ and $BA$ are both block diagonal, with the blocks of $BA$ not necessarily scalar, but with some power of them scalar. For example, if $D=\operatorname{diag}(2,-2,3,-3)$, $A$ is invertible and of the form
$$\begin{pmatrix}a & b & 0 & 0 \\ c & d & 0 & 0 \\ 0 & 0 & e & f \\ 0 & 0 & g & h\end{pmatrix}$$
then we can set $B=DA^{-1}$
Then $AB\neq BA$ but $(AB)^{2}=(BA)^{2}$. I assert that every example where $A$ is invertible and $BA$ is digaonalizable is conjugate to an example of this form.
There are similar examples one can produce when $BA$ isn't diagonalizable but has eigenvalues whose ratio is a root of unity, but I don't yet feel confident that I have a complete characterization in this case.
If neither $A$ nor $B$ is invertible, a necessary condition is that the eventual images and eventual kernels of $AB$ and $BA$ coincide, at which point I believe the problem boils down to whether the invertible parts of $AB$ and $BA$ satisfy the condition, although I have not fully verified this.