Let $A_{m\times n},B_{n\times m}$ be matrices, where $n<2m$ and $\text{rank}(A)=\text{rank}(B)=m$. Prove: $ABA\neq O$

Question

The problem is in the title. My attempt:

Let's assume by contradiction that $ABA=O$. Therefore we can conclude that the columns of $BA$ are solutions of $Ax=0$. Let $P$ be the solution space of $Ax=0$ and let $W^C_{BA}$ be the column space of $BA$. We obtained $W^C_{BA}\subseteq P$ and therefore $\text{rank}(BA)\le \text{dim}P$. We also know that $\text{dim}P=n-\text{rank}(A)=n-m$.

And now I'm stuck. Please don't provide the full solution, but rather helpful hints, thanks. Also, I can't use Sylvester's inequality.

Edit: The question already has a good answer. However, I'm wondering if the statement is still true even if we are not given that $n<2m$.

Answer 1

Since the author modified the OP,

"Without using Sylvester’s rank inequality"

Assume $ABA=0$, then we get $$\text{col}A\subseteq\ker AB$$

by Rank Theorem, $\dim\ker AB=m-\text{rank}AB,$ we have

$$\dim\text{col}A=\text{rank}A=m\le m-\text{rank}AB$$

we get

$$\text{rank}AB\le0\Longrightarrow \text{rank}AB=0\Longrightarrow AB=0\Longrightarrow \text{col}B\subseteq\ker A$$

By Rank Theorem $$\dim \text{col}B=\text{rank}B\le n-\text{rank}A$$

hence

$$m\le n-m\Longrightarrow 2m\le n$$

contradicts with $2m>n$

Answer 2

Assume $ABA=0$, then we get $$\text{col}A\subseteq\ker AB$$

by rank theorem, $\dim\ker AB=m-\text{rank}AB,$ we have

$$\dim\text{col}A=\text{rank}A=m\le m-\text{rank}AB$$

we get

$$\text{rank}AB\le0\Longrightarrow \text{rank}AB=0$$

Next, by Sylvester’s rank inequality

$$\text{rank} A+\text{rank}B-n\le \text{rank}AB$$

We get:

$$2m-n\le \text{rank}AB=0\Longrightarrow 2m\le n$$

which contradicts with $n<2m$.