If $\det(A+B)=\det(A+2B)=\det(A+3B)=1$ and $AB=BA$ then $B^2=0$

Question

Prove that if $\det(A+B)=\det(A+2B)=\det(A+3B)=1$ and $AB=BA$ then $B^2=0$.

A problem from a math competition.

$A$, $B$ are 2 by 2 complex matrices. I've tried using Cayley Hamilton theorem, on $A+B$ and $A+3B$, and combine the relations but I get the determinant of $B$ in terms of a sum of $A$ and $B$ with coeffiecients in terms of the traces of $A$ and $B$. I am trying to prove that $\det B=\mathop{\mathrm{trace}}B=0$.

As a side question to the main problem, are $A,B$ simultaneously upper triangulable?(later edit: If they are, then I found a solution, in which case, a different solution would be nice)

Answer 1

Since $\det(A+tB)=1$ for $t=1,2,3,$ $\det(A+tB)\equiv 1$ for any $t$. This implies $\det(A)=1$ and $\det(B)=0$.

Moreover, $$\begin{array}{ll} 1&=\det(A+tB)=\det[A(I+tA^{-1}B)]=\det(A)\det(I+tA^{-1}B)\\ &=\det(I+tA^{-1}B)\\ &=1+t\cdot\mathop{tr}(A^{-1}B) +t^2 \det(A^{-1}B)\end{array}$$ for all $t$.

It follows that $\mathop{tr}(A^{-1}B)=0$ and $\det(A^{-1}B)=0$. By the Cayley-Hamilton, the $2\times 2$ matrix $A^{-1}B$ satisfies $(A^{-1}B)^2=0$.

Since $AB=BA$, we have $(A^{-1}B)^2=A^{-2}B^2$. Thus, $A^{-2}B^2=0$.

Answer 2

Yes, two $2 \times 2$ matrices $A$, $B$ that commute are simultaneously upper triangulable. This is easy to see by considering the possible Jordan canonical forms of $A$:

1. If $A = \pmatrix{a & 0\cr 0 & b\cr}$ with $a \ne b$, any matrix that commutes with $A$ is diagonal.
2. If $A$ is a scalar multiple of $I$, it suffices to make $B$ upper triangular.
3. If $A = \pmatrix{a & b\cr 0 & a\cr}$ with $b \ne 0$, any matrix that commutes with $A$ is upper triangular.