In this post norm denotes a matrix norm, i.e. it is sub-multiplicative. All matrices are real. $A$ is of size $n \times k$ with independent columns ($k \leq n$). $B$ is of size $k \times n$.
Let $\| \cdot \|_\square$ be some arbitrary matrix norm. Assume $\| AB\|_\square \leq 1$. Then we know $\rho(AB) \leq 1$. Then we know from this post that $\rho(BA) \leq 1$.
Edit: thanks to @loupblanc I realized this was incorrect. From this we know that there must be a matrix norm $\| \cdot \|_\triangle$ such that $\| BA\|_\triangle \leq 1$. . In fact we only know that there is a norm which is arbitrarily close to the spectral radius.
I am interested in whether there is an easy way of defining $\| \cdot \|_\triangle$ from $\| \cdot \|_\square$. For example, is it true that $\| \cdot \|_\triangle = \|F (\cdot) Q\|_\square$ for some choice of $F$ and $Q$?
Edit: when I say the norm $\| \cdot \|_\square$ is sub-multiplicative I mean it is sub-multiplicative in the space of $n \times n$ matrices. I do not mean $\| A B \|_\square \leq \|A\|_\square \| B \|_\square$, which indeed doesn't make sense. What I mean that if, say, one decomposed $AB$ into a product of square matrices, i.e. $AB = C_1C_2$, where $C_1,C_2$ are square, then we would have $\| A B\|_\square = \| C_1 C_2 \|_\square \leq \| C_1 \|_\square \| C_2 \|_\square $.
Submultiplicative norms for non-square matrices obviously do not make sense, but with square matrices, I don't think what you originally wrote was wrong. More specifically, we have the following
If $\rho(AB)<1$, the proposition is obvious because $\rho(AB)=\rho(BA)$ and the spectral radius is the infinum of all submultiplicative matrix norms. So, it suffices to consider only the case $\rho(AB)=\rho(BA)=\|AB\|_\square=1$, and we need to find some norm such that $\|BA\|_\triangle=1$.
As $\rho(AB)=1$, we have $1 = \left(\rho(AB)\right)^k = \rho((AB)^k) \le \|(AB)^k\|_\square \le \|AB\|_\square^k = 1$ and in turn $\|(AB)^k\|_\square=1$ for all natural number $k$. Hence $AB$ is power bounded. As $(BA)^{k+1}=B(AB)^kA$, the product $BA$ and in turn its Jordan form are power bounded too. Since all matrix norms (submultiplicative or not) on $M_n(\mathbb C)$ are equivalent, the Frobenius norms of the powers of the Jordan form of $BA$ are uniformly bounded. It follows that for each unit eigenvalue of $BA$, its algebraic and geometric multiplicities coincide. That is, in the Jordan decomposition $BA=PJP^{-1}$, we may assume that $$ J=\pmatrix{T&0\\ 0&D}, $$ where $T$ is a direct sum of (upper triangular) Jordan blocks for eigenvalues whose magnitudes are strictly smaller than $1$, and $D$ is a diagonal matrix whose diagonal entries have unit moduli.
Now suppose the size of $T$ is $m\times m$. Let $\epsilon>0$ and $\Lambda=\operatorname{diag}(1,\epsilon,\epsilon^2,\ldots,\epsilon^{m-1})\oplus I_{n-m}$. When $\epsilon$ is sufficiently small, the superdiagonal of $\Lambda^{-1}J\Lambda$ can be made arbitrarily small and the diagonal entries remain invariant. Therefore the spectral norm $\|\Lambda^{-1}J\Lambda\|_2=\sigma(\Lambda^{-1}J\Lambda)$ is equal to $1$. Consequently, if we define $$ \|X\|_\triangle = \|\Lambda^{-1}P^{-1}XP\Lambda\|_2, $$ we get $\|BA\|_\triangle=1$.