Why the variance of a sum equals to sum of covariance?

Question

According to wiki:

$$\text{Var}\left( \sum_{i=1}^n X_i \right) = \sum_{i=1}^n \sum_{j=1}^n \text{Cov}(X_i, X_j)$$

I can't figure out why it's true ? (I can't prove it)

1. How can we prove it ?
2. Is there a simple example which will help me to understand this equation ?

Answer 1

This can be proved using the definition of variance and covariance.
Using the definition of variance and the linearity of the expected value you have $$Var\left(\sum_{i=1}^n X_i\right)=\mathbb{E}\left[\left(\sum_{i=1}^nX_i-\mathbb{E}\left[\sum_{i=1}^nX_i\right]\right)^2\right] = \mathbb{E}\left[\left(\sum_{i=1}^n(X_i-\mathbb{E}\left[X_i\right])\right)^2\right].$$ Let's define $a_i=X_i-\mathbb{E}\left[X_i\right].$ Expanding the right hand side of the previous equation you get $$\mathbb{E}\left[\left(\sum_{i=1}^na_i\right)^2\right]=\mathbb{E}\left[\sum_{i=1}^n\sum_{j=1}^n a_ia_j\right]=\sum_{i=1}^n\sum_{j=1}^n\mathbb{E}\left[ (X_i-\mathbb{E}\left[X_i\right])(X_j-\mathbb{E}\left[X_j\right])\right],$$ which is what you were after.
I am not sure if you can view the equality in a more intuitive way, but the result comes from the the linearity of the expected value and the expansion of the square of a sum.
Maybe it helps to be a bit more explicit in the case $n=2$. $$Var\left(X_1 + X_2\right)=\mathbb{E}\left[\left(X_1 + X_2-\mathbb{E}\left[X_1+X_2\right]\right)^2\right]=\mathbb{E}\left[\left(X_1 + X_2-\mathbb{E}\left[X_1\right]-\left[X_2\right]\right)^2\right].$$ Now you have to expand the square of the sum. One way to go about it is expanding it term by term and trying to see if you can play with the expression you obtain.
Alternatively, it can be noticed that $X_1 -\mathbb{E}\left[X_1\right]$ is the same term that appears in the definition of $Var(X_1)$, so maybe it is not such a bad idea to write the last term as $$\mathbb{E}\left[\left((X_1-\mathbb{E}\left[X_1\right])+(X_2-\left[X_2\right])\right)^2\right].$$ Expanding the square now you can easily guess/get the result.

Answer 2

I like to think of the covariance as a symmetric bilinear form. Indeed, for $X,Y,Z\in L^2(\Omega)$ and $\lambda \in \mathbb{R}$

• $\mathrm{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]=\mathbb{E}[(Y-\mathbb{E}[Y])(X-\mathbb{E}[X])]=\mathrm{Cov}(Y,X)$.
• $\mathrm{Cov}(X+\lambda Y,Z)=\mathbb{E}[(X+\lambda Y-\mathbb{E}[X+\lambda Y])(Z-\mathbb{E}[Z])]=\mathbb{E}[(X-\mathbb{E}[X])(Z-\mathbb{E}[Z])+\lambda(Y-\mathbb{E}[Y])(Z-\mathbb{E}[Z])]=\mathrm{Cov}(X,Z)+\lambda\mathrm{Cov}(Y,Z)$

by linearity of the expected value. Obviously $\mathrm{Var}(X)=\mathrm{Cov}(X,X)$. Using all those facts leads to $$ \mathrm{Var}\left(\sum_{j=1}^nX_j\right)=\mathrm{Cov}\left(\sum_{i=1}^nX_i,\sum_{j=1}^nX_j \right)=\sum_{i=1}^n\mathrm{Cov}\left(X_i,\sum_{j=1}^nX_j\right)=\sum_{i=1}^n\sum_{j=1}^n\mathrm{Cov}(X_i,X_j).$$