Covariance of Binomials

Question

I'm doing some basic error, but I just can't see where...

Let $X_i\sim \mathrm{Bin}(\theta_i,n)$, and $X_j\sim \mathrm{Bin}(\theta_j,n)$

I want to find $\mathrm{Cov}(X_i,X_j)$.

So,

$E(X_iX_j)=E(E(X_iX_j|X_i=x_i))=E(X_i n \theta_j)=n\theta_i n\theta_j$. But this isn't correct is it? Why?

Answer 1

Recall that $X=(X_i)_i$ multinomial $(n,(\theta_i)_i)$ with $\sum\limits_i\theta_i=1$ can be realized as $$X_i=\sum\limits_{k=1}^n\mathbf 1_{U_k=i},$$ for every $i$, where $(U_k)$ is i.i.d. with $P(U_k=i)=\theta_i$ for every $k$ and every $i$. Thus, for every $i\ne j$, $$ X_iX_j=\sum_k\mathbf 1_{U_k=i}\mathbf 1_{U_k=j}+\sum_{k\ne\ell}\mathbf 1_{U_k=i}\mathbf 1_{U_\ell=j}.$$ Since $[U_k=i]\cap[U_k=j]=\varnothing$ for every $k$ and the events $[U_k=i]$ and $[U_\ell=j]$ are independent for every $k\ne\ell$, this yields $$ E(X_iX_j)=\sum_{k\ne\ell}P(U_k=i)P(U_\ell=j)=n(n-1)\theta_i\theta_j. $$ Since $E(X_i)=n\theta_i$ and $E(X_j)=n\theta_j$, one gets $$ \mathrm{Cov}(X_i,X_j)=n(n-1)\theta_i\theta_j-n\theta_in\theta_j=-n\theta_i\theta_j.$$

Answer 2

The covariance should be $E(X_iX_j - E(X_i)E(X_j))$, which in your calculations could be $0$ when $i \not = j$.

But you seem to have said $E(X_j|X_i=x_i)= n\theta_j$. Are you sure? What is $E(X_j|X_i=n)$?

Answer 3

From a comment by the OP, the real question is as follows. We repeat an experiment independently $n$ times. The possible outcomes are $A_1, A_2, \dots, A_q$ with probabilities $\theta_1,\theta_2,\dots,\theta_q$. Let $X_i$ be the number of Type $A_i$ outcomes in the $n$ trials. We want $\text{Cov}(X_i,X_j)$. Informally, the covariance is negative: if $A_i$ occurs lotsa times, then probably $X_j$ will not occur many times.

So we want $E(X_iX_j)-E(X_i)E(X_j)$. Te expectations $E(X_i)$ and $E(X_j)$ are easy. We need to find $E(X_iX_j)$.

Let $U_k=1$ if on the $k$-th trial we get $A_i$. Let $V_k$ be defined similarly for $A_j$. We want $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))$. By the linearity of expectation, and the fact that $U_sV_s=0$ and $\Pr(U_sV_t=1)=\theta_i\theta_j$ if $s\ne t$, we get that $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))=n(n-1) \theta_i\theta_j$.

But $E(X_i)=n\theta_i$, with a similar expression for $E(X_j)$. Subtract. We get the desired result.