$\mathrm{rank}(AB-BA)=1$ implies $A$ and $B$ are simultaneously triangularisable

Question

Let $A$ and $B$ in $M_n(\mathbb C)$ such that the rank of $AB-BA$ is $1$. Prove that $A$ and $B$ are simultaneously triangularisable.

This generalizes the classical case $AB = BA$.

By induction on $n$, it suffices to show that $A$ and $B$ have a common eigenvector. So, it would be sufficient to find a eigenspace of $A$ which is stable by $B$ since matrices are complex.

Do you have ideas for that? Thank you.

Answer 1

Lets prove that $A$ and $B$ have at least one common eigenvector. If $x\in \ker(A-\lambda I)$, we have $$ (AB-BA)x = (A-\lambda I)Bx. $$ For $x\in \ker (AB-BA)$ this imply $Bx\in \ker(A - \lambda I)$, which means that $\ker(A-\lambda I)$ is an invariant subspace for operator $B$, so there is an eigenvector $y$ of $B$ lying in it.

We win when at least one eigenvector of $A$ or $B$ lying in $\ker(AB-BA)$. Consider the case when all the eigenvectors of $A$ and $B$ are not in $\ker(AB-BA)$. From well-known identity $$ \dim\ker(AB-BA) = n - \mathrm{rank}\,(AB-BA) = n-1, $$ so eigenvectors of both $A$ and $B$ lying in the subspace of dimension $1$, which means they have common eigenvector.

---

Now, lets $v$ be a common eigenvector of $A$ and $B$ in some basis $\{e_1,\dots, e_n\}$. Then in basis $\{v,e_2,\dots, e_n\}$ matrices of $A$ and $B$ become $$ A = \begin{pmatrix} \lambda & X \\ 0 & A_1 \end{pmatrix}, \quad B = \begin{pmatrix} \mu & Y \\ 0 & B_1 \end{pmatrix}. $$ This yields $\mathrm{rank}(A_1B_1-B_1A_1) \leq \mathrm{rank}(AB-BA)$, so we have the same situation for $A_1$ and $B_1$. Repeating this reasoning $n$ times we obtain the upper-triangular forms of $A$ and $B$ in the same basis.

Answer 2

That follows from the book Simultaneous Triangularization by Radjavi and Rosenthal (page 8). The original proof is due to Thomas Laffey.

Let $\{y\}$ be a basis of $\mathrm{Im}(AB-BA)$. Let $\lambda\in\mathrm{Spec}(B)$. If $B=\lambda I$, then there is almost nothing to do. Otherwise $F=\ker(B-\lambda I)$, $G=\mathrm{Im}(B-\lambda I)$ are non-trivial $B$-invariant subspaces. If we show that $F$ or $G$ is $A$-invariant, then we are the kings of oil.

Assume that $F$ is not $A$-invariant. Then there is $x$ s.t. $(B-\lambda I)x=0$, $(B-\lambda I)Ax\not= 0$. We have $$A(B-\lambda I)x-(B-\lambda I)Ax=ABx-BAx=-(B-\lambda I)Ax\in\mathrm{Im}(AB-BA)\cap\mathrm{Im}(B-\lambda I)\setminus\{0\}.$$ Thus $y\in G$.

Let $z\in \mathbb{C}^n$. Then $A(B-\lambda I)z$ is in the form $(B-\lambda I)Az+\alpha y$. Therefore, $G$ is $A$-invariant and we are done. $\square$