Simultaneous triangularization for AB-BA

Question

Is it true in general that if $A, B \in M_{n}(\mathbb{C})$ such that $AB-BA=P(A)$ where $P \in \mathbb{C}[x]$, then $A$ and $B$ are simultanously triangularzible? I know that if $AB-BA=A^k$ for a positive integer $k$ then it is true ($Ker(A)$ is $B$-invariant and so the triangularization builds recursively). But is it true for any polynomial $P$?

Answer 1

Here is a 2-step solution. I hope that the 1st step can be simplified, but I also hope that after my edits it is clear :)

1. We can reduce to the case when $A$ is nilpotent.

Indeed, if $A$ is general and $Q(x)=\prod_{i=1}^k(x-\lambda_i)^{n_i}$ is its characteristic polynomial ($k$ is the number of roots of $Q$), we have the $A$-invariant decomposition $$\mathbb C^n =\bigoplus_{i=1}^k V_i \quad\text{where}\quad V_i:=\ker(A-\lambda_i)^{n_i}\qquad(1)$$ Moreover we can find polynomials $Q_i$ such that $Q_i(A)$ is the projector to $V_i$ in this direct sum (e.g. by decomposing $1/Q(x)$ to partial fractions and then multiplying the terms by $Q(x)$).

Let us prove that $[Q_i(A), B]=0$ for all $i=1,\dots,k$. Since $A$ commutes with $[A,B]=AB-BA=P(A)$, we have $[Q_i(A), B]=Q_i'(A)P(A)$, and thus $[Q_i(A), B]$ commutes with $A$ and therefore also with $Q_i(A)$. Since $Q_i(A)^2=Q_i(A)$ (a projector), we get $$[Q_i(A),B]=[Q_i(A)^2,B]=$$ $$=Q_i(A)\,[Q_i(A),B]+[Q_i(A),B]\,Q_i(A)= 2[Q_i(A),B]\,Q_i(A).\quad(2)$$ Since on $V_j$ we have either $Q_i(A)=0$ (if $i\neq j$) or $Q_i(A)=1$ (if $i=j$), we get from $(2)$ that on $V_j$ either $[Q_i(A),B]=0$ or $[Q_i(A),B]=2[Q_i(A),B]$, so (in both cases) $[Q_i(A),B]=0$ on $V_j$, hence (since $j$ was arbitrary) $[Q_i(A),B]=0$ on the entire space $\mathbb C^n$.

To conclude the 1st part: $[Q_i(A),B]=0$ implies that the decomposition $(1)$ is $B$-invariant. We can thus restrict ourselves to the subspaces $V_i=\ker(A-\lambda_i)^{n_i}\subset\mathbb C^n$, and if we replace $A$ by $A-\lambda_i$, we make it nilpotent there.

2. Supposing that $A$ is nilpotent.

Since $\operatorname{Tr} (AB-BA)=0$ and also $\operatorname{Tr}A^k=0$ for all $k>0$, we must have $P(0)=0$, i.e. $AB-BA = H(A)A$ for some polynomial $H$. Hence $\ker A$ is $B$-invariant, and this implies (by induction) that a simultaneous triangularization exists.