Let $A,B\in M_3(\mathbb{C})$ be invertible matrices such that $AB=BA=X$, $A^{T}+A=B^{T}+B=X^{T}+X$. Then, is $\det(X - I) = 0$?

Question

Let $A,B\in M_3(\mathbb{C})$ be invertible matrices such that $AB=BA=X$, $A^{T}+A=B^{T}+B=X^{T}+X$, then:

(A) $A=B$

(B) $\det(A-I)=0$

(C) $\det(B-I)=0$

(D) $\det(X-I)=0$

My working: $AB+(AB)^T=X+X^T\implies AB+B^TA^T=B+B^T\implies (A-I)B+B^T(A^T-I)=O_3$ $\implies (A-I)B+((A-I)B)^T=O_3\implies (A-I)B$ is skew symmetric matrix of order $3$ $\implies \det(A-I)=0$.

Similarly $\det (B-I)=0$.

How to check option D any counter example for D?

Further progress:

$A^T+A=B^T+B\iff A= \begin{pmatrix} \alpha & a_1 & a_2 \\ x-a_1 & \beta & a_3 \\ y-a_2 & z-a_3 & \gamma \end{pmatrix}$ and $B =\begin{pmatrix} \alpha & b_1 & b_2 \\ x-b_1 & \beta & b_3 \\ y-b_2 & z-b_3 & \gamma \end{pmatrix}$.

Counter example for (A):

$A=I_3$ and $B= \begin{pmatrix} 1 & b_1 & b_2 \\ -b_1 & 1 & b_3 \\ -b_2 & -b_3 & 1 \end{pmatrix}\implies AB=BA=B\implies X=B$.

We can see that all conditions meet out and $A\ne B$ for at least one $b_i\ne0$.

Answer 1

Note that two $3\times3$ skew symmetric matrices $Y$ and $Z$ commute only if they are linearly dependent. More specifically, note that \begin{aligned} YZ&:=\pmatrix{0&-y_3&y_2\\ y_3&0&-y_1\\ -y_2&y_1&0} \pmatrix{0&-z_3&z_2\\ z_3&0&-z_1\\ -z_2&z_1&0}\\ &=\pmatrix{-(y_2z_2+y_3z_3)&y_2z_1&y_3z_1\\ y_1z_2&-(y_1z_1+y_3z_3)&y_3z_2\\ y_1z_3&y_2z_3&-(y_1z_1+y_2z_2)}\\ &=\mathbf z\mathbf y^T-\mathbf y^T\mathbf zI \end{aligned} where $\mathbf y=(y_1,y_2,y_3)^T$ and $\mathbf z$ is defined analogously. (This is actually the vector triple product formula.) So, if $YZ=ZY$, we get $\mathbf z\mathbf y^T=\mathbf y\mathbf z^T$. Hence $\mathbf y$ and $\mathbf z$ are linearly dependent and $Y,Z$ are linearly dependent too.

Now return to your question. You have already shown that $Y:=(A-I)B$ and $Z:=(B-I)A$ are skew symmetric. Since $A$ and $B$ commute, so do $Y$ and $Z$. Hence $Y$ and $Z$ are linearly dependent. As $3\times3$ skew symmetric matrices are singular, we have $Yv=Zv=0$ for some nonzero vector $v$. This implies that $0\ne w:=Bv=ABv=BAv=Av$ and $Aw=w=Bw$. Thus $ABw=w$ and $X-I$ is singular.

Answer 2

Rephrasing the question, let $A, B \in \operatorname{Mat}_3(\mathbb{C})$ be two invertible matrices such that $AB = BA$, and $A - B$ and $AB - A$ are skew-symmetric. Is it true that 1 is always an eigenvalue of $AB$?

Since $A - B$ is skew-symmetric, we may write $A = Q + S$ and $B = Q + T$ for some symmetric matrix $Q$ and skew-symmetric matrices $S, T$. Applying the condition $AB = BA$ gives $$\begin{aligned} 0 &= AB - BA\\ &= (Q^2 + QT + SQ + ST) - (Q^2 + QS + TQ + TS) \\ &= \underbrace{(SQ - QS)}_{\text{symmetric}} + \underbrace{(QT - TQ)}_{\text{symmetric}} + \underbrace{(ST - TS)}_{\text{skew-symmetric}}. \end{aligned}$$ Equating parts gives $ST = TS$ and $(S-T)Q = Q(S-T)$, or in other words $S$ commutes with $T$ and $Q$ commutes with $(S-T)$.

Now looking at the next condition, we have $$\begin{aligned} AB - A &= (Q^2 + QT + SQ + ST) - (Q + S) \\ &= \underbrace{(Q^2 - Q + ST)}_{\text{symmetric}} + \underbrace{(QT + SQ - S)}_{\text{skew-symmetric}}, \end{aligned}$$ where we can see that $QT + SQ$ is skew-symmetric by rearranging $(S-T)Q=Q(S-T)$. Since $AB - A$ should be purely skew-symmetric, we find $ST = Q - Q^2$. In particular, this also shows that $Q$ commutes with $ST$.

Now suppose that one of $S$ or $T$ is zero, say $S = 0$. The equation $ST = Q - Q^2$ implies that $Q = Q^2$, and since $A = Q$ is invertible we can cancel to get $Q = I$. Then $AB = I + T$ has 1 as an eigenvalue, since we can take any vector $v$ in the kernel of $T$ (which necessarily exists, since $T$ is skew-symmetric on an odd-dimensional space) and get $ABv = (I +T)v = v$.

Suppose from now on that $S \neq 0$ and $T \neq 0$, each with eigenvalues $\{\pm s, 0\}$ and $\{\pm t, 0\}$ for $s, t \neq 0$. Since $S$ and $T$ commute, and have distinct eignevalues, they simultaneously diagonalise. We have essentially two possibilities for what these diagonalisations look like, depending on whether $\ker ST$ is one-dimensional or two-dimensional. (I'm using the shorthand $[S]$ to mean $P^{-1} S P$ for the simultaneous diagonalising matrix $P$).

Case 1: $$\begin{aligned}\, [S] = \begin{pmatrix}s & & \\ & -s & \\ & & 0 \end{pmatrix}, \quad [T] = \begin{pmatrix}t & & \\ & -t & \\ & & 0 \end{pmatrix}, \quad [ST] = \begin{pmatrix}st & & \\ & st & \\ & & 0 \end{pmatrix}, \quad [S-T] = \begin{pmatrix}s-t & & \\ & t-s & \\ & & 0 \end{pmatrix} \end{aligned}$$

Case 2: $$\begin{aligned}\, [S] = \begin{pmatrix}s & & \\ & -s & \\ & & 0 \end{pmatrix}, \quad [T] = \begin{pmatrix}t & & \\ & 0 & \\ & & -t \end{pmatrix}, \quad [ST] = \begin{pmatrix}st & & \\ & 0 & \\ & & 0 \end{pmatrix}, \quad [S-T] = \begin{pmatrix}s-t & & \\ & -s & \\ & & t \end{pmatrix} \end{aligned}$$

In case 1, let $v \in \ker ST$, necessarily we also have $Sv = Tv = 0$. Since $Q$ commutes with $ST$ and $\ker ST$ is one-dimensional, $v$ is also an eigenvector of $Q$, say $Qv = qv$. Applying $ST = Q - Q^2$ to $v$ gives $q^2 - q = 0$, and since $Sv = 0$ we have $Av = Qv \neq 0$ since $A$ is invertible. Hence $q = 1$. Then $ABv = Qv = v$, so $AB$ has 1 as an eigenvalue.

In case 2 we can't use the same approach, since a vector in $\ker ST$ is not necessarily an eigenvector of $Q$, due to the kernel being two-dimensional. However we can instead use $S - T$ to show we have an appropriate eigenvector. Since $S - T$ is skew-symmetric it must have zero as an eigenvalue, and since by assumption $s, t \neq 0$ we must have $s = t$, i.e. the top-left entry of $S - T$ is zero. This means that $S - T \neq 0$, and since it commutes with $Q$, $Q$ also simultaneously diagonalises in the same basis as $S$ and $T$. We can then choose $v \in \ker S$ again (or $\ker T$) and repeat similar logic to show that $AB$ has 1 as an eigenvalue.