Finding the Covariance Matrix of Two Random Vectors, Given Their Covariance Matrices

Question

Let's say we have two random vectors, x and y. We know their own covariance matrices, Cov(xx) and Cov(yy), as well as Mean(x) and Mean(y). How can we find the covariance matrix of x and y, Cov(xy), given what we know?

Answer 1

You cannot find the covariance matrix $\text{Cov}(x, y)$ with the given information. For example, take $x \sim N(0, I_n)$ standard $n$-dimensional Gaussian.

• Example 1. Let $y \sim N(0, I_n)$ be independent of $x$. Then the means of $x$ and $y$ are the zero vectors, and their individual covariance matrices are both $I_n$, but the covariance matrix $\text{Cov}(x,y)$ is the zero matrix.
• Example 2. Let $y=x$. Again, the means of $x$ and $y$ are the zero vectors, and their individual covariance matrices are both $I_n$. But now $\text{Cov}(x,y)$ is also the identity matrix $I_n$.