When $(A-B)^2=0_{3\times3}$, $(AB-BA)^2=0$ if and only if $\operatorname{adj}(A^2-B^2)=0$

Question

Let $A, B \in M_3(\mathbb{C})$.

a) Prove that $\operatorname{adj}(A+B)+\operatorname{adj}(A-B)=2\left(\operatorname{adj}(A)+\operatorname{adj}(B)\right)$.

b) If $(A-B)^2=0$, prove that $(AB-BA)^2=0$ if and only if $\operatorname{adj}(A^2-B^2)=0$.

The first point is not difficult, it can be solved with a simple calculus. I have problems at the second point. I am pretty sure that we use ranks. If we consider that $(AB-BA)^2=0_3$ then we have to show that $\operatorname{rank}(A^2-B^2)\leq 1$, which would be equivalent to $\operatorname{adj}(A^2-B^2)=0$, but I don't know how to do it. It is obvious that we should use point a) but I can't see how. How can be the second point solved?

Answer 1

$A-B$ is $3\times3$. If it squares to zero, either it is zero or it is similar to $e_1e_2^T$. In the former case, we have $A=B$ and the rest is trivial. In the latter case, by a change of basis, assume that $A-B=e_1e_2^T$. Then \begin{align} AB-BA&=e_1e_2^TB-Be_1e_2^T,\tag{1}\\ A^2-B^2&=e_1e_2^T+e_1e_2^TB+Be_1e_2^T.\tag{2} \end{align} It follows that the three statements below are equivalent:

• the rank of $AB-BA$ is at most one,
• $Be_1$ is a multiple of $e_1$ or $e_2^TB$ is a multiple of $e_2^T$,
• the rank of $A^2-B^2$ is at most one.

Now suppose $(AB-BA)^2=0$. Since the matrix is $3\times3$, the rank of $AB-BA$ is at most one. Therefore the rank of $A^2-B^2$ is at most one too.

Conversely, suppose the rank of $A^2-B^2$ is at most one. Then the rank of $AB-BA$ is also at most one. Yet, a traceless rank-one matrix must be nilpotent and its index of nilpotency must be $\le2$. Hence $(AB-BA)^2=0$.