Let $(A_n)$, $(B_n)$, $(C_n)$ be sequences of $n\times k$ matrices, where $k<n$ is fixed and
$$A'_nA_n=B'_nB_n=C'_nC_n=I_k \hspace{1cm} $$
$$A_n'C_n=0$$
$$\|A_n\|_F=\|B_n\|_F=\|C_n\|_F=\sqrt{k}$$
for each $n$.
How can I bound the Frobenius norm of the matrix $A_nA'_nB_nB'_nC_nC'_n$ as $n\to\infty$?
I feel it should be quite small when $n$ is large. But I don't know how to show it. Any help is greatly appreciated.
We have $$ \|AA'BB'CC'\|_F^2 = \operatorname{tr}[AA'BB'(CC')^2BB'AA'] \\ = \operatorname{tr}[(AA')^2BB'(CC')^2BB'] = \operatorname{tr}[AA'BB'CC'BB']\\ \leq \|AA'BB'\|_F \|CC'BB'\|_F, $$ and $\|AA'BB'\|_F^2 = \operatorname{tr}[AA'BB'] = \|A'B\|_F^2$.
With that established, note that the matrix $W = [A \ \ C]$ has orthogonal columns. It follows that $$ \|B\|_F^2 \geq \|W'B\|_F^2 = \left\|\pmatrix{A'B\\C'B} \right\|_F^2 = \|A'B\|_F^2 + \|C'B\|_F^2. $$ Now, we note that the maximum $$ \max_{a,b\geq 0} ab \quad \text{s.t.} \quad a^2 + b^2 = \|B\|_F^2 $$ is attained with $a = b = \|B\|_F/\sqrt{2}$ (as can be seen with the AM-GM inequality for instance). Putting all this together, we have $$ \|AA'BB'CC'\|_F^2 \leq \|A'B\|_F \cdot \|C'B\|_F \leq \frac 12 \|B\|_F^2 = k/2. $$