Two invertible matrices

Question

Let $A,B$ be two $n\times n$ invertible matrices with complex entries. Also, let $\alpha, \beta \in \mathbb{C}$ with $|\alpha| \neq |\beta|$ such that $\alpha AB+\beta BA=I_n$. Prove that $\det(AB-BA)=0$.

I tried to manipulate the given equation in order two get $(AB-BA)$ as a factor somewhere, but didn't manage to get anything useful. I also thought of using $A^{-1}$ and $B^{-1}$ somewhere, but I only got messier relations.

Answer 1

I leave my first answer below. Here is a much easier one: As below, we may assume $AB + \gamma BA = I$, where $|\gamma|\neq 1$ and $\gamma\neq 0$. Put $\lambda_0 = (1+\gamma)^{-1}$. Then $$ AB-\lambda_0 = 1 - \lambda_0 - \gamma BA = -\gamma\left(BA - \frac{1-\lambda_0}{\gamma}\right) = -\gamma(BA-\lambda_0). $$ Hence, as $AB$ and $BA$ have the same eigenvalues, $$ \sigma(BA-\lambda_0) = \sigma(AB-\lambda_0) = -\gamma\cdot\sigma(BA-\lambda_0).$$ Thus, multiplication by $(-\gamma)$ leaves the finite set $\sigma(BA-\lambda_0)$ invariant. But, as $|\gamma|\neq 1$, this is only possible if $\sigma(BA-\lambda_0) = \{0\}$. Hence, $BA-\lambda_0$ is nilpotent. And as $$ AB-BA = I - \gamma BA - BA = I - \lambda_0^{-1}BA = -\lambda_0^{-1}(BA - \lambda_0), $$ the same holds for $AB-BA$. In particular, $AB-BA$ is not invertible, i.e., $\det(AB-BA)=0$.

---

The statement is clear for $\alpha = 0$. Hence, let $\alpha\neq 0$. In this case, with $A' = \alpha A$ we have $A'B + \frac{\beta}{\alpha}BA' = I$. Hence, we may assume that $AB + \gamma BA = I$ with $|\gamma|\neq 1$ and $\gamma\neq 0$.

Let $x$ be an eigenvector of $AB$ with respect to the eigenvalue $\lambda$. Then $$ \lambda x + \gamma BAx = x, $$ that is, $$ BAx = \frac{1-\lambda}\gamma x. $$ But we know that $AB$ and $BA$ have exactly the same eigenvalues (even the same Jordan structures) as they are both invertible. Hence, the function $f(z) = \tfrac{1-z}\gamma$ is a selfmap of the set of eigenvalues. Therefore, there is an eigenvalue $\lambda$ such that $f^n(\lambda) = \lambda$ for some $n$, where $f^n = f\circ\ldots\circ f$ ($n$ times).

One can easily prove by induction that $$ f^n(z) = \frac{1 - (-\gamma^{-1})^n}{1+\gamma} + (-\gamma^{-1})^nz $$ and then (since $(-\gamma^{-1})^n\neq 1$) that the only fixed point of each $f^n$ is $z = (1+\gamma)^{-1}$. Thus, $\lambda = (1+\gamma)^{-1}$. In particular, $f(\lambda) = \lambda$. But then, with the eigenvector $x$ from above, we have $$ (BA-AB)x = BAx - ABx = f(\lambda)x - \lambda x = 0. $$ Hence, the matrix $BA-AB$ is not invertible, meaning that $\det(BA-AB) = 0$.

Answer 2

I eventually managed to find a solution, different from @amsmath's one. From the given equation we get $$AB=\frac{1}{\alpha}I_n-\frac{\beta}{\alpha}BA \quad (1)$$ and also $$\alpha B+\beta A^{-1}BA=A^{-1} \quad (2)$$ hence $$d=\det(AB-BA)=\det \left(\frac{1}{\alpha}I_n-\frac{\alpha+\beta}{\alpha}BA \right)=\frac{1}{\alpha^n}\det(I_n-(\alpha+\beta)BA)$$ But $A^{-1}A=I_n$ so $d=\frac{1}{\alpha^n}\det(A^{-1}A-(\alpha+\beta)BA)=\frac{1}{\alpha^n}\det A \det(A^{-1}-(\alpha+\beta)B)$ and using $(2)$ we'll have $$d=\frac{1}{\alpha^n}\det A \det(\beta A^{-1}BA-\beta B)=\frac{\beta^n}{\alpha^n}\det A \det(A^{-1}BA-B)=\frac{\beta^n}{\alpha^n}\det(BA-AB)$$ Finally, $d=\det(AB-BA)=\frac{\beta^n}{\alpha^n}(-1)^n\det(AB-BA)$ and since $|\alpha| \neq |\beta|$, the conclusion follows.

Answer 3

As loup blanc has pointed out in a comment, the assumption that $A$ and $B$ are invertible is useless. Even the assumption that $|\alpha|\ne|\beta|$ is too strong. As we will see shortly, all we need is that $\alpha^n\ne(-\beta)^n$.

From the condition that $\alpha AB+\beta BA=I$, we get \begin{align} \alpha(AB-BA) &= (I-\beta BA)-\alpha BA = I - (\alpha+\beta)BA,\\ -\beta(AB-BA) &= -\beta AB + (I-\alpha AB) = I - (\alpha+\beta)AB. \end{align} Since $A$ and $B$ are square matrices, $AB$ and $BA$ have identical spectra. It follows that $\alpha^n\det(AB-BA) = (-\beta)^n \det(AB-BA)$. Therefore, when $\alpha^n\ne(-\beta)^n$, we must have $\det(AB-BA)=0$.