I want a correct and feasible answer to this question. So does anyone have any creative ideas to prove this equation?
$A$ and $B$ are $3\times3$ matrices.
$\det(AB - BA) = \dfrac{1}{3}\operatorname{Trace}\left((AB - BA)^3\right)$
Answer:
We can write and compute both sides to prove it but this is not a good solution!
On
Choose a basis which puts $AB-BA$ in Jordan normal form (both $\det$ and $\text{Tr}$ are invariant under basis change, so this is allowed). Then since a commutator is traceless, the diagonal must have the form $(a,b,-a-b)$. Then $\det(AB-BA)=-ab(a+b)$, and $$\frac{1}{3}\text{Tr}(AB-BA)^3=\frac{1}{3}(a^3+b^3-(a+b)^3) = -ab(a+b)$$ as desired.
On
Wikipedia gives the following identity for the determinant of a $k\times k$ matrix: $$\det A= \frac{1}{k!} \det\begin{pmatrix} \operatorname{tr}A & k-1 &0&\cdots & \\ \operatorname{tr}A^2 &\operatorname{tr}A& k-2 &\cdots & \\ \vdots & \vdots & & \ddots & \vdots \\ \operatorname{tr}A^{k-1} &\operatorname{tr}A^{k-2}& & \cdots & 1 \\ \operatorname{tr}A^k &\operatorname{tr}A^{k-1}& & \cdots & \operatorname{tr}A \end{pmatrix}. $$ In particular when $k=3$, and using $AB-BA$ in place of $A$, we have that $$ \det (AB-BA)=\frac{1}{3!}\det\begin{pmatrix} 0 & 2 &0 \\ \operatorname{tr}(AB-BA)^2 &0& 1 \\ \operatorname{tr}(AB-BA)^{3} &\operatorname{tr}(AB-BA)^{2}& 0 \\ \end{pmatrix}. $$ Expanding the determinant along the top row yields your formula.
On
Another proof. It is known that for an arbitrary $3 \times 3$ matrix $M$, we have $$ \det(M) = \frac 16\left[ \operatorname{tr}^3(M) + 2 \operatorname{tr}(M^3) -3\operatorname{tr}(M)\operatorname{tr}(M^2)\right] $$ Setting $M = AB - BA$, we note that $\operatorname{tr}(M) = 0$ so that the above simplifies to $$ \det(M) = \frac 13 \operatorname{tr}(M^3) $$ which is the formula that we wanted.
This follows easily from Cayley-Hamilton theorem. Since $M=AB-BA$ has zero trace, by Cayley-Hamilton theorem, $M^3=cM+dI_3$ where $c$ is some scalar and $d=\det(M)$. Therefore $\operatorname{tr}(M^3)=3d$ and the result follows.