Let $A$ and $B$ be nilpotent matrices that commute with $[A,B]$. If $A$, $B$, and $[A,B]$ are all nilpotent, show that $A+B$ is nilpotent.

Question

How to prove that $A + B$ is nilpotent, when $A$, $B$, $[A, B]$ are nilpotent matrices, and also $A$ and $[A, B]$, $B$ and $[A, B]$ are two pairs of commuting matrices?

Looks like I should use binomial formula for commuting matrices, but if $$A [A, B] = [A, B] A \\ B [A, B] = [A, B] B,$$ does it mean that $A$ and $B$ commute?

EDIT: Let $C = [A, B]$, so $AB = BA + C$, now we need to prove that every product of $A$ and $B$ is $C^kB^mA^n$, if $k, m, n > 0$. If it will be done, we can say, that with big $N$ (for example $2(k + m + n)$) every of the $2^N$ addends of the $(A + B)^N$ is equal to $0$. So the matrix will be nilpotent. How can i prove this?

Answer 1

We work in an arbitrary unital ring $R$ (you can take $R$ to be a matrix ring if you want). Recall that $[X,Y]=XY-YX$ for all $X,Y\in R$. Let $I\in R$ be the multiplicative identity. We use the convention $X^0:=I$ for any $X\in R$.

Suppose that $A$ and $B$ are elements in $R$ such that both $A$ and $B$ commute with $C:=[A,B]$. (I do not yet assume that $A$, $B$, and $C$ are nilpotent.) We claim that any product $X_1X_2\cdots X_n$, where $X_i\in \{A,B\}$ for $i=1,2,\ldots,n$, is a $\mathbb{Z}$-linear combination of $A^p B^q C^r$, where $p,q,r\in\mathbb{Z}_{\geq 0}$ are such that $$\frac{n}2\leq p+q+r\leq n\,.$$ (You can also prove that $p+q+2r=n$, but this is not important in the case at hand.) The basis step $n=1$ is trivial (you may also include the case $n=0$, as the empty product is $I=A^0B^0C^0$).

Suppose that $n>1$. If $X_n=B$, then we apply the induction hypothesis on $X_1X_2\cdots X_{n-1}$. Thus, $X_1X_2\cdots X_{n-1}$ is a linear combination of $A^{p'}B^{q'}C^{r'}$, where $p',q',r'\in\mathbb{Z}_{\geq 0}$ satisfy $$\frac{n-1}{2}\leq p'+q'+r'\leq n-1\,.$$ Since $B$ commutes with $C$, multplying each term $A^{p'}B^{q'}C^{r'}$ on the right yields $A^pB^qC^r$ with $p:=p'$, $q:=q'+1$, and $r:=r'$. That is, $p+q+r=p'+q'+r'+1$, making $$\frac{n}{2}<\frac{n+1}{2}\leq p+q+r\leq n\,.$$ We now suppose that $X_n=A$. If $X_1=X_2=\ldots=X_n=A$, then $X_1X_2\cdots X_n=A^n$ is in the required form. From now on, we assume that $X_k=B$ for some $k\in\{1,2,\ldots,n-1\}$. We take the largest possible value of such $k$, so $X_{k+1}=X_{k+2}=\ldots=X_n=A$.

Now, note that $X_kX_{k+1}X_{k+2}\cdots X_n=BA^{n-k}=A^{n-k}B-[A^{n-k},B]$. It is an easy exercise to show that $$[A^m,B]=m\,A^{m-1}C$$ for all $m=1,2,3,\ldots$. That is, $$X_1X_2\cdots X_n = X_1X_2\cdots X_{k-1}A^{n-k}B-X_1X_2\cdots X_{k-1}\big((n-k)\,A^{n-k-1}C)\,.$$ The first term $X_1X_2\cdots X_{k-1}A^{n-k}B$ is handled by the paragraph above. The second term is equal to $-(n-k)\,X_1X_2\cdots X_{k-1} A^{n-k-1}C$. We use the induction hypothesis on $X_1X_2\cdots X_{k-1}A^{n-k-1}$, so $X_1X_2\cdots X_{k-1}A^{n-k-1}$ is a linear combination of $A^{p'}B^{q'}C^{r'}$ for some $p',q',r'\in\mathbb{Z}_{\geq 0}$ such that $$\frac{n-2}{2}\leq p'+q'+r'\leq n-2\,.$$ Thus, $X_1X_2\cdots X_{k-1} A^{n-k-1}C$ is a linar combination of $A^pB^qC^r$ with $p:=p'$, $q:=q'$, and $r:=r'+1$, and $p+q+r=p'+q'+r'+1$, so $$\frac{n}{2}\leq p+q+r\leq n-1<n\,.$$

The claim has now been established. In fact, it follows that the unitary subalgebra $S$ of $R$ generated by $A$ and $B$ is spanned (over $\mathbb{Z}$) by elements of the form $A^pB^qC^r$.

Finally, if $A$, $B$, and $C$ are nilpotent, say $A^a$, $B^b$, and $C^c$ are zero for some $a,b,c\in\mathbb{Z}_{>0}$. Pick $n\in\mathbb{Z}$ such that $n\geq 2a+2b+2c-5$. Therefore, $(A+B)^n$ is a $\mathbb{Z}$-linear combination of $A^pB^qC^r$, where $p,q,r\in\mathbb{Z}_{\geq 0}$ satisfy $$a+b+c-3<\frac{n}{2}\leq p+q+r \leq n\,.$$ Thus, $p>a-1$, $q>b-1$, or $r>c-1$ for each occurrence of $A^pB^qC^r$. This proves that $(A+B)^n=0$ for every integer $n\geq 2a+2b+2c-3$.

Answer 2

When $A,B\in M_n(\mathbb{C})$ commute with $[A,B]$, one says that $A,B$ quasi commute; this property implies that $A,B$ are simultaneously triangularizable over $\mathbb{C}$ (McCoy). Thus, we may assume that $A,B$ are upper triangular. That implies, in particular, that $[A,B]$ is nilpotent.

Now, we assume moreover that $A,B$ are nilpotent (note that the hypothesis $[A,B]$ nilpotent is useless).

Then $A,B$ are strictly upper triangular and $A+B$ too, that implies that $A+B$ is nilpotent.

EDIT. Answer to the OP.

More generally, let $G$ be a Lie algebra over $\mathbb{C}$ with bracket $[.,.]$ and let $A,B\in G$ s.t. $C=[A,B]\not= 0$ and $[A,C]=[B,C]=0$.

Then $\mathrm{span}(A,B)$, the Lie algebra generated by $A,B$, has dimension $3$ and is nilpotent; then it is solvable.

If, moreover, $G$ is finite dimensional, then it is a matrix Lie algebra (Ado); thus, applying the Lie theorem, we deduce that $A,B$ are simultaneously triangularizable in the following sense: there is $\phi$, a one to one representation of $\mathrm{span}(A,B)$ in $M_n(\mathbb{C})$, s.t. $\phi(A),\phi(B)$ are simultaneously triangularizable.