Multiplication of matrix: ABAB

Question

I was doing this Putnam exercise:

Let $A$, $B$ be $n \times n$ matrices such that $ABAB = 0$. Can we conclude that $BABA = 0$?

I realized that this is not always true, it can fail to non-invertible matrices with an arbitrary n order. But, when I searched in google to confirm my answer, I read an interesting answer from another user here that says: "If $n=2$ and $AB$ is nilpotent, so as the $BA$ and this affirmation is always true"

Okay, it does not disprove my answer but it was interesting to know

For $n = 2$: The characteristic polynomials of $AB$ and $BA$ are the same (for any $n$). Therefore, if $AB$ is nilpotent then $BA$ is also nilpotent. So, we necessarily have $BABA = (BA)^2 = 0$.

I can see the characteristic polynomial are equal by the equation $t^2 - \operatorname{tr}(AB)t + \det(AB)$, but what does this imply? I mean, my knowledge about characteristic polynomials is restricted to find the eigenvalues and eigenvectors.

Answer 1

Since ${\rm trace}(AB) = {\rm trace} (BA)$ and $\det (AB)=\det (BA)$ it means that for $n=2$ matrices $AB$ and $BA$ have the same characteristic polynomial so they have the same eigenvalues.

But in this case you don't need that. Since $ABAB = 0$ we get $BABABA =0$ so $(BA)^3=0$ which means that $BA$ is also nilpotent but with order $3$ or less, so it eigenvalues are all $0$.

Answer 2

I don't see why the characteristic polynomial should be involved here.

Instead note that $(BA)^3=B(AB)^2A=0$ so $BA$ is indeed nilpotent, although possibly only for the third power. When $n=2$ any nilpotent matrix may be reduced to the form $$ M = \pmatrix{ 0 & 1 \\ 0 & 0}, $$ so the order of milpotency must be 2. You may construct counterexamples for $n\geq 3$, e.g.: $$ A = \pmatrix{ 0 & 1 &0\\ 0 &0&0\\ 0 & 0 &1}, \ \ B = \pmatrix{ 1 & 0 &0\\ 0 &0&1\\ 0 & 0 &0}, \ \ $$ for which you check that $(AB)^2=0$ but $(BA)^2\neq 0$.