Let $X\in\mathcal{M}_{n\times p}(\mathbb{R})$ with rank(X)=k and let $\epsilon \sim N(0,I_p)$ be a vector with i.i.d. Gaussian components.

48 Views Asked by At

Let $X$ be an $n \times p$ matrix with rank$(X)=k$. Let $\epsilon \sim N(0,I_p)$ be a vector with the i.i.d. Gaussian components.

(a) Show that $$E\|\ X\epsilon \|_2^2 = \|\ X \|_F^2 = \sum_{j=1}^k \sigma_j^2$$ where $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_k > 0$ are positive signular values of $X$ and $\|\ X \|_F$ is the Frobenius norm of $X$.

(b) Show that if $X$ is a projection matrix, i.e. $X=A(A^TA)^{-1}A^T$ for some matrix $A : n \times p$ with rank$(A)=p$, then $$E \|\ X\epsilon \|_2^2 = \sigma^2p.$$

By the singular value decomposition theorem we know that $X$ can be decomposed in the form: $$X=U\Sigma V^H,$$ where $U$ is an $n \times n$ unitary matrix, $\Sigma$ is a diagonal $n \times p$ matrix with non-negative real numbers on the diagonal, and $V$ is an $p\times p$ unitary matrix, and $V^H$ denotes the conjugate transpose of $V$. The diagonal entries $\sigma_i$ of $\Sigma$ are known as the singular values of $M$, in other words $\Sigma$ has $\left\{\sigma_1, \dots, \sigma_k\right\}$ on the diagonal, as rank$(X)=k$. Now, we have that from definition of Frobenius norm, \begin{align*} \|\ X \|_F^2 &= \operatorname{Tr}(XX^H) \\ &= \operatorname{Tr}((U\Sigma V^H)(U\Sigma V^H)^H) \\ &= \operatorname{Tr}(U\Sigma V^H V \Sigma^H U^H) \\ &= \operatorname{Tr}(U\Sigma\Sigma^H U^H) \\ &= \operatorname{Tr}(\Sigma\Sigma^H)\\ &= \sum_{j=1}^k \sigma_j^2. \end{align*} since $V^HV=I$ and $U^HU=I$, where we also utilized the cyclic property of trace. All that remains to be shown is that $E\|\ X\epsilon \|_2^2 = \sum_{j=1}^k \sigma_j^2$. I am not sure about this part of (a) and, hoping that once I figure that part out of (a), I'll have a better idea of (b). Any help would be appreciated!