Probability of tossing marbles,covariance

Question

Suppose that we have $n$ marbles and toss them in $k$ boxes. $p_i$ is the probability that the marble lands in $\#i$-th box,$i=1,...,k$ and $\sum_{i=1}^kp_i=1$. Suppose that $Y_l$ is $1$ if $l$-th marble lands in the $\#i$-th box, and $0$ otherwise. Similarly $Z_l$ is a random variable equal to $1$ iff $l$-th marble lands in the box number $j$.Let $X_i=\sum_{l=1}^n Y_l$ and $X_j=\sum_{l=1}^n Z_l$. How do I compute $Cov(X_i,X_j)$ for $i\neq j$ immediately from this formula $Cov(X_i,X_j)=E[(X_i-E[X_i])(X_j-E[X_j])].$ I do not want to use this consequence $Cov(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j].$ This is because this simplified formula does not contain $X_i$ directly, but only through $E$,and I can justify that $E[X_iX_j]=0$ but I do not know what should I substitute for $X_i$ or $X_j$ in the original formula $E[(X_i-E[X_i])(X_j-E[X_j])].$ The result should be $Cov(X_i,X_j)=-np_ip_j.$

Answer 1

First remark that the $l$th ball landing in the $i$th box is independent from the $k$th ball landing in the $j$th box, as long as $l\neq k$. Therefore

$$\text{Cov}(Y_l,Z_k)=0 \; \text{, if } \; l\neq k .$$

However, when we consider $\text{Cov}(Y_l,Z_l)$ something else happens: if the $l$th ball lands in the $i$th box, it can not land in any of the other boxes. So there is dependency. You can have the following three situations

$$\begin{cases}Y_l=1 \; \text{ and } \; Z_l=0 & \text{ with probability } \; p_i \\ Y_l=0 \; \text{ and } \; Z_l=1 & \text{ with probability } \; p_j \\ Y_l=0 \; \text{ and } \; Z_l=0 & \text{ with probability } \; 1-p_i-p_j \end{cases}$$

The last case corresponds to when the ball lands in another box than the $i$th or $j$th.

You can also easily determine that $\text{E}[Y_l]=p_i$ and $\text{E}[Z_l]=p_j$ so that we can now turn to the computation of the covariance

$$\text{Cov}(Y_l,Z_l) = (1-p_i)(0-p_j)p_i + (0-p_i)(1-p_j)p_j + (0-p_i)(0-p_j)(1-p_i-p_j) = -p_ip_j$$

Now, turning our attention to $\text{Cov}(X_i,X_j)$

$$\text{Cov}(X_i,X_j)=\text{Cov}(\sum_{l=1}^n Y_l,\sum_{k=1}^n Z_k) = \sum_{l=1}^n\sum_{k=1}^n\text{Cov}( Y_l, Z_k) = \sum_{l=1}^n \text{Cov}( Y_l, Z_l) = -n p_i p_j \; .$$

Answer 2

I agree with @owen88. It's virtually always the case that the simplified expression for the covariance will get you to your answer faster and easier. This is more of a comment but it's easier to write math in the answer section. So, in case the theory behind the result isn't clear, note that by definition, we have $$Cov(X_i, X_j)=E[(X_i-E[X_i])(X_j-E[X_j])]$$ The term inside the expectation is a random variable and we can expand it as follows: $$E[X_iX_j-X_iE[X_j]-X_jE[X_i]+E[X_i]E[X_j]]$$ The most important fact of the expectation operator is that it is linear. Thus, we can rewrite this as $$E[X_iX_j]-E[X_iE[X_j]]-E[X_jE[X_i]]+E[E[X_i]E[X_j]]$$ We then note that $E[X_i]$ and $E[X_j]$ are just constants. Since the EV of a constant times a random variable is equal to the constant times the EV of the random variable, we can rearrange to get $$E[X_iX_j]-E[X_i]E[X_j]-E[X_i]E[X_j]+E[X_i]E[X_j]E[1]$$ Simplification then yields the result of interest: $$Cov(X_i, X_j)=E[X_iX_j]-E[X_i]E[X_j]$$