How does $\sum^n_{i=1}\sum^n_{j=1}x_ix_j\mathrm{Cov}\left(Z_i,Z_j\right)$ equal to $\mathrm{Cov}\left(\sum^n_{i=1}x_iZ_i, \sum^n_{j=1}x_jZ_j\right)$?

Question

I have a proof which I don't understand quite one step:

Let $\sum$ be a covariance matrix. Then $\sum$ is non-negative definite.

Proof:

Let $Y=x_1Z_1+x_2Z_2+...+x_nZ_n$. We want to prove $\underline{x}^T\sum\underline{x} \geq0$ for all $\underline{x} \in \Bbb R^n\backslash\{0\}$

Then

$$\underline{x}^T\sum\underline{x}=\sum^n_{i=1}\sum^n_{j=1}x_ix_j\operatorname{Cov}(Z_i,Z_j)=\operatorname{Cov}(\sum^n_{i=1}x_iZ_i, \sum^n_{j=1}x_jZ_j)=\operatorname{Cov}(Y,Y)=\operatorname{ Var}(Y)\geq0$$

Therefore, $\sum$ is n.n.d.

I don't quite understand the part where $$\sum^n_{i=1}\sum^n_{j=1}x_ix_j\operatorname{Cov}\left(Z_i,Z_j\right)=\operatorname{Cov}\left(\sum^n_{i=1}x_iZ_i, \sum^n_{j=1}x_jZ_j\right)$$

I know the formula $\operatorname{Cov}(au+bv,w)=a\operatorname{Cov}(u,w)+b\operatorname{Cov}(v,w)$

So, to understand how how they derived it, I tried to derive it myself by expanding $\sum^n_{i=1}\sum^n_{j=1}\operatorname{Cov}\left(Z_i,Z_j\right)$:

$$x_1x_1\operatorname{Cov}(Z_1,Z_1)+x_1x_2\operatorname{Cov}(Z_1,Z_2)+x_1x_3\operatorname{Cov}(Z_1,Z_3)+...+x_1x_n\operatorname{Cov}(Z_1,Z_n)+$$

$$+x_2x_1\operatorname{Cov}(Z_2,Z_1)+x_2x_2\operatorname{Cov}(Z_2,Z_2)+x_2x_3\operatorname{Cov}(Z_2,Z_3)+...+x_2x_n\operatorname{Cov}(Z_2,Z_n)+$$

$$+$$

$$.$$

$$.$$

$$+$$

$$+x_nx_1\operatorname{Cov}(Z_n,Z_1)+...+x_nx_n\operatorname{Cov}(Z_n,Z_n)$$

So, $$\underbrace{x_1x_1}_{a}\operatorname{Cov}(\underbrace{Z_1}_{w},\underbrace{Z_1}_{u})+\underbrace{x_1x_2}_{b}\operatorname{Cov}(\underbrace{Z_1}_{w},\underbrace{Z_2}_{v})=\operatorname{Cov}(x_1x_2Z_1+x_1x_2Z_2,Z_1)$$

Ok so I iterate:

$$\underbrace{1}_{a}\operatorname{Cov}(\underbrace{x_1x_2Z_1+x_1x_2Z_2}_{u},\underbrace{Z_1}_{w})+\underbrace{x_1x_3}_{b}\operatorname{Cov}(\underbrace{Z_1}_{w},\underbrace{Z_3}_{v})=\operatorname{Cov}(x_1x_1Z_1+x_1x_2Z_2+x_1x_3Z_3,Z_1)$$

$$.$$

$$.$$

$$\operatorname{Cov}(x_1(\sum^n_{i=1}x_iZ_i),Z_1)$$

So the whole sum becomes $$\sum^n_{i=1}\operatorname{Cov}\left(x_i\left(\sum^n_{j=1}x_jZ_j\right),Z_i\right)$$

So how does this equal to $$\operatorname{Cov}\left(\sum^n_{i=1}x_iZ_i, \sum^n_{j=1}x_jZ_j\right)?$$

Answer 1

We indeed have:

$$\sum_{i=1}^{n}\mathsf{Cov}\left(x_{i}\left(\sum_{j=1}^{n}x_{j}Z_{j}\right),Z_{i}\right)=\sum_{i=1}^{n}x_{i}\mathsf{Cov}\left(\sum_{j=1}^{n}x_{j}Z_{j},Z_{i}\right)=\sum_{i=1}^{n}\mathsf{Cov}\left(\sum_{j=1}^{n}x_{j}Z_{j},x_{i}Z_{i}\right)=$$$$\mathsf{Cov}\left(\sum_{j=1}^{n}x_{j}Z_{j},\sum_{i=1}^{n}x_{i}Z_{i}\right)$$

as you wondered in the last lines of your question.

But your deduction is quite cumbersome.

Bilinearity of covariance tells us that: $$\sum_{i=1}^{n}\sum_{j=1}^{n}\mathsf{Cov}\left(X_{i},X_{j}\right)=\mathsf{Cov}\left(\sum_{i=1}^{n}X_{i},\sum_{j=1}^{n}X_{j}\right)$$

Applying this on $X_{i}=x_{i}Z_{i}$ we get:$$\sum_{i=1}^{n}\sum_{j=1}^{n}\mathsf{Cov}\left(x_{i}Z_{i},x_{j}Z_{j}\right)=\mathsf{Cov}\left(\sum_{i=1}^{n}x_{i}Z_{i},\sum_{j=1}^{n}x_{j}Z_{j}\right)$$

Here we have $\mathsf{Cov}\left(x_{i}Z_{i},x_{j}Z_{j}\right)=x_{i}x_{j}\mathsf{Cov}\left(Z_{i},Z_{j}\right)$ so that we can also write: $$\sum_{i=1}^{n}\sum_{j=1}^{n}x_{i}x_{j}\mathsf{Cov}\left(Z_{i},Z_{j}\right)=\mathsf{Cov}\left(\sum_{i=1}^{n}x_{i}Z_{i},\sum_{j=1}^{n}x_{j}Z_{j}\right)$$

Answer 2

Here is a short version of this proof: \begin{align} x^{\top}\Sigma x&=\mathsf{E}[x^{\top}(Z-\mathsf{E}Z)(Z-\mathsf{E}Z)^{\top}x] \\ &=\mathsf{E}[Z^{\top}x-\mathsf{E}Z^{\top}x]^2=\operatorname{Var}(Z^{\top}x). \end{align}