Product of nilpotent matrices invertible

Question

Do there exist a positive integer $n$ and two nilpotent $n\times n$ matrices $A$ and $B$ such that $AB$ is invertible?

If yes, can you give me an example, if no, can you prove why?

Answer 1

It suffices to show that the determinate of $AB$ is zero. We have that $det(AB)=det(A)det(B)$, so it suffices to show that $det(A)=0$. But $0=det(0)=det(A^n)=det(A)^n$ for $n>0$ by nilpotence, so indeed we have $det(AB)=0$ and no such combination exists.

Answer 2

The key idea one needs is the following: if $AB$ is invertible, then both $A$ and $B$ are. Indeed, if $(AB)C=C(AB)=I$, then $A(BC)=I$, which implies that $A$ is invertible (for matrices, it is enough to check invertibility on one side). Similarly, $(CA)B=I$ implies that $B$ is invertible.

If $A$ is nilpotent, then $A$ is not invertible: if $n$ is the smallest exponent with $A^n=0$, then if $A$ were invertible we have $0=A^n=A\,A^{n-1}$, and multiplying by $A^{-1}$ on the left we get $A^{n-1}=0$.

So if $A$ and $B$ are nilpotent, none of them is invertible. And so $AB$ cannot be invertible.

Answer 3

If $B$ is nilpotent, then there is a non-zero vector $x$ such that $Bx=0$. So $ABx=0$ has a non-trivial solution $x$.