How to prove the following rank equality?

Question

Let $A,B\in M_{n}$. Show that $$\mbox{rank}\,(A-ABA)=\mbox{rank}\,(A)+\mbox{rank}\,(I_{n}-BA)-n$$

I have shown one part by using Sylvester rank inequality. How to prove the equality case?

Answer 1

Fix a basis of $\mathbb F^n$ (where $\mathbb F$ is the field of entries of $A$ and $B$) so that $A$ and $B$ can be considered as linear transformations. Now, since $A - ABA = A(I - BA)$, we have $\require{AMScd}\DeclareMathOperator{\im}{im}\DeclareMathOperator{\rank}{rank}\DeclareMathOperator{\null}{null}$

\begin{CD} \mathbb F^n @>I-BA>> \im(I - BA) @>A>> \im(A - ABA)\\ \end{CD}

so that by the rank-nullity theorem applied on the second map, $A$,

$$\dim \im(I - BA) = \dim \im(A - ABA) + \null A$$

where $\null A$ is the nullity of $A$ in $\im(I - BA)$. Thus,

\begin{equation} \rank(I - BA) = \rank(A - ABA) + \null A. \tag{1}\label{rk-null} \end{equation}

We shall show that the nullity of $A$ in $\im(I - BA)$ is the nullity of $A$ in $\mathbb F^n$ itself.

Let $x \in \ker_{\mathbb F^n} A$. Then $Ax = 0 \implies BAx = 0$, so we may write $x = x - BAx = (I - BA)x$, which shows that $\ker_{\mathbb F^n} A \subseteq \im(I - BA)$. Thus, $\ker_{\im(I - BA)} A = \ker_{\mathbb F^n} \cap \im(I - BA) = \ker_{\mathbb F^n} A$. Thus, $\null A = n - \rank A$ in $\im(I - BA)$.

Then the result follows from \eqref{rk-null}.