How can I prove this theorem for the maximum singular value of a matrix $A$?

Question

Let $A \in \mathbb C^{p \times q}$ and $B \in \mathbb C^{q \times p}$. Let $\bar\sigma(A)$ be the largest singular value of $A$, and let $\rho(AB)$ be the spectral radius of $AB$, which is the maximum of the absolute values of the eigenvalues of $AB$. Suppose that $I - AB$ is singular, such that $\text{det}(I - AB) = 0$. How can I show that $$ \max_{\bar\sigma(B) \leq 1} \rho(AB) = \bar\sigma(A) $$ So far, I was able to show that $$ \min_{B \in \mathbb C^{q \times p}} \left\{\bar\sigma(B) \mid \text{$I - AB$ is singular}\right\} = \frac{1}{\bar\sigma(A)} $$ If the above theorem is true, which I know it is, then $$ \frac{1}{\displaystyle\max_{\bar\sigma(B) \leq 1} \rho(AB)} = \min_{B \in \mathbb C^{q \times p}} \left\{\bar\sigma(B) \mid \text{$I - AB$ is singular}\right\} $$ but I'm not sure how to proceed any further.

Answer 1

You cannot prove it because it is false. Counterexamples:

1. Suppose $\|A\|_2<1$. Note that it is in general true that $$ \rho(AB)\le\|AB\|_2\le\|A\|_2\|B\|_2\le\|A\|_2\quad\text{whenever $\|B\|_2\le1$,}\tag{$\ast$} $$ Therefore, if $\|A\|_2<1$ and $\|B\|_2\le1$, we will have $\rho(AB)<1$ and hence $I-AB$ is nonsingular. Therefore $\max_{\|B\|_2\le1,\,\det(I-AB)=0}\rho(AB)$ is undefined because no $B$ satisfies the given constraints.
2. Suppose $A=kuu^\ast$ for some $k>1$ and for some unit vector $u$. The set $\left\{B:\,\|B\|_2\le1\text{ and }I-AB\text{ is singular}\right\}$ is non-empty in this case (take e.g. $B=\frac{1}{k}I$). However, as the rank of $AB$ is at most one, $I-AB$ is singular if and only if the spectrum of $AB$ is $\{1,0,\ldots,0\}$. It follows that $\rho(AB)=1<k=\|A\|_2$.

Edit. If you do not require $I-AB$ to be singular, it is true that $\max_{\|B\|_2\le1}\rho(AB)=\|A\|_2$. In this case, as shown in $(\ast)$, $\rho(AB)$ is always bounded above by $\|A\|_2$. Hence it suffices to show that $\rho(AB)=\|A\|_2$ for some $\|B\|_2=1$. Let $u$ and $v$ be two unit vectors such that $Av=\|A\|_2u$. That is, $v$ is a right singular vector corresponding to the largest singular value of $A$. Let $B=vu^\ast$. Then $\|B\|_2=1$ and $\rho(AB)=\rho(Avu^\ast)=\rho(\|A\|_2uu^\ast)=\|A\|_2$.