Matrices $AB$ and $BA$ identity

Question

We have $n$ by $n$ matrices $A, B$ with complex entries. We denote $X=AB$ and $Y=BA$. We have to prove that $\det(I+X+X^2+X^3)=\det(I+Y+Y^2+Y^3)$.

I showed that we have the following equality: $AB(I+X+X^2+X^3)=A(I+Y+Y^2+Y^3)B$

So, if $\det(AB)\neq 0$ we get our result. What can I do for the case when $\det(AB)= 0$ ? Could it be something using the fact that $\det(X^4-I)=\det(X-I)\det(I+X+X^2+X^3)$ and that $\det(X-I)=\det(Y-I)$ ?

Answer 1

Well, you can use the following magic: $$\det(I+X+X^2+X^3)-\det(I+Y+Y^2+Y^3)$$ is just a polynomial of coefficients in $X$ and $Y$. Invertible matrices form an open subset in all matrices, and since you proved that this polynomial equals $0$ on that subset, it also equals $0$ for all matrices.

P.S. I think there is a theorem saying the same, but I sadly can't recall the name of it.

Answer 2

It's sufficient to do it for the case $\det(AB)\ne0$. For $\det(AB)=0$ take sequences $A_n\to A$ and $B_n\to B$ with $\det(A_nB_n)\ne0$ and take limits.