Problem: Let $A,B$ be two $3\times 3$ matrices with complex entries, such that $BA^2=A^2B$. Prove that $\det(AB-BA)=0$
attempt:
$A(AB-BA)=A^2B-ABA=BA^2-ABA=(BA-AB)A=-(AB-BA)A$
So
$\det[(A(AB-BA)]=(-1)^3\det[(AB-BA)A]\Rightarrow |A||AB-BA|=0$. Thus, if $|A| \not=0$;
I can say $|AB-BA|=0$.
another approach:
$3\det(AB-BA)=\text{trace}[(AB-BA)^3]$ So we need to prove that $\text{trace}[(AB-BA)^3]=0$ i.e: let $AB=X,BA=Y$ then $$\text{trace}[(AB-BA)^3]=\text{trace}(X-Y)^3\\ =\text{trace}(X^3-X^2Y-XYX-YX^2+XY^2+YXY+Y^2X-Y^3)=0\Leftrightarrow \\ \text{trace}(X^3+3XY^2)=\text{trace}(Y^3+3YX^2)$$.
but $$\text{trace}(XY^2)=\text{trace}(ABBA.BA)=\text{trace}(BA.ABBA)=\text{trace}(AABBBA)=\text{trace}(AAABBB)$$
and $$\text{trace}(YX^2)=\text{trace}(B.AABAB)=\text{trace}(AABAB.B)=\text{trace}(BAAABB)=\text{trace}(AAAABBB)$$
So we need to prove that $\text{trace}(ABABAB)=\text{trace}(BABABA) ?????$