$A, B$ are $3 \times 3$ matrices such that $(A - B)^2 = 0$. Prove that $\operatorname{Tr}(AB - BA)^3 = 0$.

Question

I have been trying to solve this recent linear algebra problem:

Let $A, B$ be $3 \times 3$ matrices such that $(A-B)^2 = 0$. Prove that $\det (AB - BA) = 0$.

This was my approach:$\DeclareMathOperator{\Tr}{Tr}$

The following equality holds for any $3\times 3$ matrices $A, B$:

$$\det (AB - BA) = \frac13 \Tr(AB - BA)^3$$

It follows from the Hamilton-Cayley theorem applied on $AB - BA$. Therefore, it suffices to prove that $\Tr(AB - BA)^3 = 0$.

Expanding gives that $\Tr(AB - BA)^3$ is equal to:

\begin{align} \Tr\left(\color{magenta}{ABABAB} - \color{blue}{ABABBA} - \color{purple}{ABBAAB} + \color{purple}{ABBABA} - \color{blue}{BAABAB} + \color{olive}{BAABBA} + \color{olive}{BABAAB} - \color{magenta}{BABABA}\right)\\ \end{align}

where the same-colored terms are cyclic permutations of each other so have the same trace.

So, $$\Tr(AB - BA)^3 = 2\Tr BAABBA - 2\Tr BAABAB = 2 \Tr BAAB(BA - AB)$$

I figured this was a good place to try to use the assumption $(A - B)^2 = 0$:

$$0 = (A - B)^2 = A^2 + B^2 - AB - BA \implies A^2 = AB + BA - B^2$$

So we have:

$$BAAB(BA - AB) = B(AB + BA - B^2)B(AB - BA) = $$ $$\color{OrangeRed}{BABBBA} + \color{green}{BBABBA} - BBBBBA - \color{green}{BABBAB} - \color{OrangeRed}{BBABAB} + BBBBAB$$

Again, the same-colored terms cancel out when taking the trace so:

$$\Tr BAAB(BA - AB) = \Tr BBBB(AB - BA)$$

A possible development is:

\begin{align}2 \Tr BAAB(BA - AB) &= \Tr (AB-BA)(BBBB - BAAB)\\ &= \Tr (AB-BA)(BBBB - BAAB)\\ &= \Tr (AB-BA)B(B^2 - A^2)B \end{align}

But using $A^2 + B^2 = AB + BA$ here again gives $\Tr (BA - AB)BAAB$, so nothing new.

Is there a way to finish the proof?

Answer 1

Forget $A$ and $B$ are $3\times 3$ matrices. Let's say $A, B$ are $n \times n$ matrices satisfying the algebraic relation $(A-B)^2 = 0$. Let $X = A-B$, then $X^2 = 0$ and

$$AB-BA = (B+X)B - B(X+B) = XB-BX$$ Let $P = XB = (A-B)B$ and $Q = BX = B(A-B)$, we have $$QP = BXXB = B(A-B)^2B = 0$$ For any integer $m > 1$, when we expand $(AB-BA)^m = (P-Q)^m$ into sum of monomials in $P,Q$. If $Q$ appear before any $P$, then that term vanishes. As a result, only terms that all $P$ is on the left of $Q$ survives. More precisely,

$$(AB-BA)^m = \sum_{\ell=0}^m (-1)^\ell P^{m-\ell} Q^\ell = P^m - P^{m-1}Q + P^{m-2}Q^2 + \cdots + (-1)^m Q^m$$

For terms that contains both $P$ and $Q$ (i.e $0 < \ell < m$), we have

$${\rm Tr}(P^{m-\ell}Q^\ell) = {\rm Tr}(Q^\ell P^{m-\ell}) = {\rm Tr}(Q^{\ell-1}(QP)P^{m-\ell-1}) = {\rm Tr}(Q^{\ell-1} 0_n P^{m-\ell-1}) = {\rm Tr}(0_n) = 0$$ This leads to

$${\rm Tr}(AB-BA)^m = {\rm Tr}(P^m + (-1)^mQ^m)$$

When $m$ is odd, this becomes

$${\rm Tr}((AB-BA)^m) = {\rm Tr}((XB)^m - (BX)^m) = {\rm Tr}(X(BX)^{m-1}B - (BX)^{m-1}BX)\\ = {\rm Tr}([X,(BX)^{m-1}B]) = 0$$

When $m = 3$, this reduces to the identity you wish to show.

Answer 2

Let $N=A-B$. Then $N^2=0$ and $C=AB-BA=NB-BN$. Therefore \begin{align} C^2&=(NB-BN)(NB-BN)=NBNB-NBBN+BNBN,\\ C^3&=(NB-BN)(NBNB-NBBN+BNBN)\\ &=NBNBNB-NBNBBN+NBBNBN-BNBNBN.\tag{1} \end{align} Now, using the cyclic property of matrix trace $\operatorname{tr}(XY)=\operatorname{tr}(YX)$ and the assumption that $N^2=0$, we immediately see that the two middle summands on line $(1)$ are traceless and the traces of the two outer summands cancel out each other. Hence $\operatorname{tr}(C^3)=0$.