What is the intuition behind a low-rank covariance matrix?

Question

Let $X, Y$ be random vectors. Let $K = \text{Cov}(X,Y) = E[XY^T] - E[X]E[Y]^T$ be the covariance matrix of $X$ and $Y$. Assume $K$ is low rank.

I'm trying to come up with simple intuitive examples about what having a low rank covariance matrix means but I'm having trouble. I understand that a low rank matrix means most of the column vectors are linearly dependent on other column vectors, and I understand that the covariance matrix shows the variance relationships between each random variable. But from here I'm having trouble coming up with an intuitive explanation or example of where this would be useful.

Answer 1

One way this matrix can be low-rank is if $X$ and $Y$ are linear functions of the same random vector $Z$, which is itself much lower-dimensional. For simplicity, suppose that $X$ and $Y$ are mean zero, $n$-dimensional random vectors, so that $K=\mathbb{E}[XY^T]$. As an example, let $A,B\in \mathbb{R}^{n\times k}$, and let $Z\sim \mathcal{N}(0,I_k)$ be a standard $k$-dimensional Gaussian, where $k\ll n$. Then let $X=AZ$ and $Y=BZ$. In this case, $X$ and $Y$ depend linearly on the same randomness, so we would guess the covariance rank to be of order $k$, not $n$. Indeed, we find that \begin{equation} K=\mathbb{E}[XY^T]=\mathbb{E}[AZZ^TB^T]=A\mathbb{E}[ZZ^T]B^T=AB^T. \end{equation} Notice that $AB^T$ is rank at most $k$, so is low-rank.

Answer 2

My intuition behind low-rank covariance is that $X$ and $Y$ live in a lower-dimensional manifold.

You can visualize this in 2D by starting with a full-rank covariance and progressively bringing one of the eigenvalues to 0. You will observe that samples will all be contained in a 1D line.