How to show for any orthogonal vector set and any matrix $\sum_u \|Au\|_2^2 \leq \|A\|_F^2$?

76 Views Asked by At

The following paper on page 16, in line 17 Online Principal Component Analysis says for any orthogonal vector set and any matrix $\sum_u \|Au\|_2^2 \leq \|A\|_F^2$ is true. However, if we let $u_i$'s in $\mathbb{R}^m$ be a set of orthogonal vectors and $A \in \mathbb{R}^{n \times m}$ we have

$$\sum_u \|Au\|_2^2=\|AU\|_F^2$$ where $U$ is the matrix that has $u_i$ as its columns and $\|\cdot\|_F$ is Frobenius norm. Using Cauchy-Schwarz inequality

$$\|AU\|_F^2 \leq \|A\|_F^2\|U\|_F^2$$

My question is how we can ignore $\|U\|_F^2$?

1

There are 1 best solutions below

0
On

Note that $A^*A$ is a square matrix and $A^*A \ge 0$ so there exists an orthonormal basis $\{u_1, \ldots, u_m\}$ for $\mathbb{R}^m$ such that $A^*A u_i = \lambda_i u_i$ for some $\lambda \ge 0$.

We have $$\sum_{i=1}^m \|Au_i\|_2^2 = \sum_{i=1}^m \langle Au_i, Au_i\rangle = \sum_{i=1}^m \langle A^*Au_i, u_i\rangle = \sum_{i=1}^m \lambda_i =\operatorname{Tr}(A^*A) = \|A\|_F^2$$

The interesting part is that the sum $\sum_{i=1}^m \|Au_i\|_2^2$ is actually independent of the choice of the orthonormal basis $\{u_1, \ldots, u_m\}$. Indeed, if $\{v_1, \ldots, v_m\}$ is some other orthonormal basis for $\mathbb{R}^m$, we have \begin{align} \sum_{i=1}^m \|Au_i\|_2^2 &= \sum_{i=1}^m \langle A^*Au_i, u_i\rangle\\ &= \sum_{i=1}^m \left\langle \sum_{j=1}^m\langle u_i,v_j\rangle A^*A v_j , \sum_{k=1}^m\langle u_i,v_k\rangle v_k\right\rangle\\ &= \sum_{j=1}^m \sum_{k=1}^m \left(\sum_{i=1}^m\langle u_i,v_j\rangle \langle v_k,u_i\rangle\right)\langle A^*A v_j,v_k\rangle\\ &= \sum_{j=1}^m \sum_{k=1}^m \langle v_j,v_k\rangle\langle A^*A v_j,v_k\rangle\\ &= \sum_{j=1}^m \langle A^*A v_j,v_j\rangle\\ &= \sum_{j=1}^m \|Av_j\|_2^2 \end{align}

Hence we can extend any orthonormal set $\{v_1, \ldots, v_k\} \subseteq \mathbb{R}^m$ to an orthonormal basis $\{v_1, \ldots, v_m\}$ for $\mathbb{R}^m$ to obtain $$\sum_{i=1}^k \|Av_i\|_2^2 \le\sum_{i=1}^m \|Av_i\|_2^2 = \sum_{i=1}^m \|Au_i\|_2^2 = \|A\|_F^2 $$