What is the variance of X A?

Question

Let $\mathbf{X}$ be a random matrix of dimension $n\times p$.

Let $\mathbf{A}$ be a constant matrix of dimension $p\times p$

What is $\text{Var}\left(\mathbf{XA}\right)$?

---

Context : Generate multivariate normal data with an eigenvalue decomposition.

• Generate $\mathbf{Z}$, an $n\times p$ matrix of independent standard normal samples
• Given a desired covariance matrix $\mathbf{\Sigma}_{p\times p}$, apply the eigenvalue decomposition $\mathbf{\Sigma} = \mathbf{Q\Lambda Q}^T = \mathbf{Q\Lambda}^{1/2}\mathbf{\Lambda}^{1/2} \mathbf{Q}^T$
• Let $\mathbf{A} = \mathbf{\Lambda}^{1/2} \mathbf{Q}^{T}$
• I was hoping for a simple formula like below so that $\text{Var}(\mathbf{Z A^T}) = \mathbf{A}\text{Var}(\mathbf{Z}) \mathbf{A^T} = \mathbf{\Sigma}$.

A similar approach that doesn't work.

If we seek $\text{Var}\left(\mathbf{BX}\right)$ for constant matrix $\mathbf{B}$ with suitable dimension then the result becomes $$\text{Var}\left(\mathbf{BX}\right) = \text{E} \left((\mathbf{BX} - \text{E}(\mathbf{BX})(\mathbf{BX} - E(\mathbf{BX}))^T\right) \\ =\text{E} \left(\mathbf{B}(\mathbf{X} - \text{E}(\mathbf{X})(\mathbf{X} - E(\mathbf{X}))^T\mathbf{B}^T\right) \\ =\mathbf{B}\,\text{E}\left((\mathbf{X} - \text{E}(\mathbf{X})(\mathbf{X} - E(\mathbf{X}))^T\right)\mathbf{B}^T \\ =\mathbf{B}\,\text{Var}(\mathbf{X})\mathbf{B}^T \\ $$ But for $\text{Var}(\mathbf{XA})$ the $\mathbf{A}$ and $\mathbf{A}^T$ matrices are in the middle of the product, so they can not be factored out. What to do?

Answer 1

You're applying the eigenvalue decomp backwards. Instead of defining $\mathbf{X}:=\mathbf {ZA}$ with $\mathbf{A} := \mathbf{\Lambda}^{1/2} \mathbf{Q}^{T}$, do it the other way: Define $\mathbf X:=\mathbf {AZ}$ with $\mathbf{A}:=\mathbf{Q}\mathbf{\Lambda}^{1/2} $. With this choice you can apply the formula you want: $$\operatorname{Var}\mathbf {X}=\operatorname{Var}\mathbf {AZ}=\mathbf {A }(\operatorname{Var}\mathbf{Z})\mathbf{A}^T=\mathbf{AIA}^T=\mathbf{AA}^T=\mathbf{Q\Lambda Q}^T=\mathbf{\Sigma}. $$

Answer 2

You are writing an expression for the covariances of the rows rather than of the columns. Are you sure that's what you want to be doing?

The short answer is that if you are taking covariances of columns and $A$ expresses linear combinations of columns, the text-book like formula works. It also works if you are taking covariances of rows and linear combinations of rows. (Why you would do this I have no idea.) If you are taking covariances of rows and linear combinations of columns, I can't see why there should be a simple formula.