Let $X$ and $Y$ be two matrices different from $I$ such that $XY=YX$ and $X^n-Y^n$ is invertible for some natural number $n$ . If $X^n-Y^n=X^{n+1}-Y^{n+1}=X^{n+2}-Y^{n+2}$, then:
A) $I-X$ is singular
B) $I-X$ is singular
C) $X+Y=XY+I$
D) $(I-X)(I-Y)$ is non-singular
Like everyone, I started off taking $n=1$.
Now $|X-Y|\ne 0$ (because it's invertible)
then $X-Y=(X-Y)(X+Y)=(X-Y)(X^2+Y^2+XY)$
making it $(X-Y)(X-Y-I)=O$ and $(X-Y)(X^2+Y^2+XY-I)=O$
and after this. I'm lost and require help.
On
We assume that $X,Y\in M_m$.
-- Proposition. There are an invertible $R\in M_m$ and $p<m$ s.t. $R^{-1}XR=diag(I_p,V_{m-p}),R^{-1}YR=diag(P_p,I_{m-p})$ where $P^n=0_p,V^n=0_{m-p}$.
Proof. i) Since $X,Y$ commute, they are simultaneously triangularizable; we may assume that they are triangular and let $(\lambda_i)$, $(\mu_i)$ be their diagonals. Since $X+Y=I+XY$, $\lambda_i+\mu_i=1+\lambda_i\mu_i$ and, consequently, $\lambda_i=1$ OR $\mu_i=1$. Moreover, $X^n-Y^n=X^{n+1}-Y^{n+1}$ are invertible implies that $\lambda_i^n-\mu_i^n=\lambda_i^{n+1}-\mu_i^{n+1}\not= 0$; finally,
$$(*)\text{ one of }\lambda_i,\mu_i\text{ is }1\text{ and the other is }0$$ and the eigenvalues of $X,Y$ are in $\{0,1\}$.
ii) Then we may assume that $X=diag(I_p+U,V_{m-p})$ where $U,V$ are nilpotent. Since $XY=YX$ and, according to $(*)$, $Y$ is in the form $Y=diag(P_p,I_{m-p}+Q)$ where $P,Q$ are nilpotent. Moreover, $(I+U)^n-P^n=(I+U)^{n+1}-P^{n+1}$, that implies $rank(P^n-P^{n+1})=rank(P^n)$ is equal to $rank((I+nU+\cdots)-(I+(n+1)U+\cdots))=rank(U)$. In the same way, $X^{n+1}-Y^{n+1}=X^{n+2}-Y^{n+2}$ implies $rank(P^{n+1})=rank(U)$; we deduce that $rank(P^n)=rank(P^{n+1})$, that implies $P^n=0$ and, therefore, $U=0$. In the same way, we obtain $V^n=0,Q=0$. $\square$
Our assumptions are as follows:
$1.) \: X,Y \neq I;$
$2.) \:X \: \text{and} \:Y \:\text{commute;}$
$3.) \:\text{there is some }n \text{ such that }X^n - Y^n \:\text{is invertible;}$
$4.) \: \text{for that same }n \text{, we have that } X^n - Y^n = X^{n+1} - Y^{n+1} = X^{n+2}-Y^{n+2}.$
$\text{--} $
By 2.), it is easily verifiable that $X^jY^k = Y^kX^j $, for any $j,k \in \Bbb{N}. $
Then, for the $n$ in 3.), we have
$\begin{align} \smash{(X^{n+1} - Y^{n+1})(X+Y)}&=X^{n+2} + X^{n+1}Y - Y^{n+1}X-Y^{n+2}\\ &=X^{n+1}Y - Y^{n+1}X + X^{n+2} - Y^{n+2}\\ &=X^nXY - Y^{n}XY + X^{n+2} - Y^{n+2}\\ &=(X^n - Y^n)XY + (X^{n+2} - Y^{n+2})\\ \end{align}$
By 4.), we replace in the above LHS and RHS to achieve
$(X^n - Y^n)(X +Y) = (X^n - Y^n)XY + (X^n - Y^n)$
Since $X^n - Y^n$ invertible, we can left-side multiply both sides by whatever that inverse is to get
$X+Y = XY + I $
Hence, C) is true.
$\text{--} $
From that, we now rearrange and have
$\begin{align} \smash{0}&=X+Y - XY - I\\ &=X-XY + Y-I\\ &=X(I-Y)-(I-Y)\\ &=(X-I)(I-Y)\\ \end{align}$
Multiply both sides by $-1$, and we have
$ (I-X)(I-Y) = 0$,
which is not invertible. Hence, D) is false.
$\text{--} $
Since $Y \neq I $, then $ I-Y \neq 0$. This means that $I-Y$ doesn't send everything to $0$;
i.e. there is some $u,v \neq 0 $ such that
$ (I-Y)u=v$.
Left-side multiply each side by $ (I-X)$ and we get
$(I-X)(I-Y)u = (I-X)v$
But $(I-X)(I-Y)=0$, so $ (I-X)v = 0 $, meaning there is a non-zero $v$ such that
$(I-X)v = 0v$.
Thus, $ 0$ is an eigenvalue of $ (I-X)$, and since the determinant of an operator is the product of its eigenvalues, then
$ \det(I-X) = 0 \implies (I-X) $ is singular $\implies $A) is true.
$\text{--} $
I assume B) from your problems was meant to read "$ I-Y$ is singular", and by the commutativity of $X$ and $Y$, the above argument can be reworked to show just that; i.e. B) is true.