Matrices such that $A^2+B^2=2AB$ have a nilpotent commutator

Question

Let $A$ and $B$ be two $n \times n$ non-commuting matrices with complex elements such that $A^2 + B^2 = 2AB$. Prove that there is at least one $m \leq \frac{n+1}{2}$ so that $(AB-BA)^m = O$, where $O$ is the null matrix.

Answer 1

Indeed, @Mosquite's comment is extremely helpful. The main idea is to use the argument in Show that a matrix is nilpotent. Let $C=AB-BA = (A-B)^2$ and we will prove that $\mathrm{tr}(C^k)=0$ for $k\geq 1$.

From $C=(A-B)^2$, we see that $C$ commutes with $A-B$. Thus, $C^k$ commutes with $A-B$ for any $k\geq 1$.

Since $\mathrm{tr}(AB)=\mathrm{tr}(BA)$, we have $\mathrm{tr}(C)=0$. Suppose that for some $k\geq 2$ we proved that $\mathrm{tr}(C^{k-1})=0$. Then by $\mathrm{tr} (XY)= \mathrm{tr} (YX)$, we have $$ \begin{align} \mathrm{tr}(C^k)&= \mathrm{tr} C^{k-1}( AB-BA) \\ &= \mathrm{tr} C^{k-1}( (A-B)B+B^2-BA)\\ &=\mathrm{tr} ( (A-B) C^{k-1} B + C^{k-1}B(B-A) )\\ &=\mathrm{tr}( C^{k-1}B(A-B) + C^{k-1}B(B-A)) =0. \end{align} $$

Therefore, the claim that $\mathrm{tr}(C^k)=0$ for $k\geq 1$ is proved. Consequently, $C$ is nilpotent so that $A-B$ is also nilpotent. Thus, $(A-B)^n=O$. Then for a least integer $m$ with $2m\geq n$, we have $$ (AB-BA)^m = (A-B)^{2m} = O. $$ This $m$ satisfies $m\leq \frac{n+1}2$. So, we are done.