Could you check my results for entropies?

Question

Could you please check my results to the following exericse:

Let $X,Y$ be independent RVs, each with values in $\{0, 1\}$ and probabilities $(1/2,1/2)$. Compute $I(X; Y )$, $I(X + Y ; X)$ and $I(X + Y ; X − Y )$.

Remark: $I$ stand for the mutual information of $X$ and $Y$. It holds $$I(X;Y) = H(X)+H(Y)-H(X;Y)$$ , where $H$ denotes the entropy.

With this in mind I computed the resprective entropies as:

$$H(X) = H(Y) = 1$$ $$H(X+Y) = H(X-Y) =1.5$$ $$H(X,Y) = 2$$ $$H(X+Y, X) = H(X+Y, Y) = 2.5$$ $$H(X+Y, X-Y) = 3$$

In the end I get $I(X; Y ) = I(X + Y ; X) = I(X + Y ; X − Y ) = 0$.

Is this O.K.?

Answer 1

Some sanity checks that we can do without getting into the full calculation:

• $H(X+Y,X)$ and $H(X+Y,X-Y)$ should both be at most $2$, because $H(X,Y)$ is $2$, and the pairs $(X+Y,X)$ and $(X+Y,X-Y)$ shouldn't have more entropy than $(X,Y)$: after all, we can compute them from $(X,Y)$!
• In fact, knowing $X+Y$ and $X$, we can compute $Y$, and knowing $X+Y$ and $X-Y$, we can compute $X$ and $Y$. So we should have $H(X+Y,X) = H(X+Y,X-Y) = H(X,Y) = 2$.
• $I(X+Y;X)$ should be positive because some values of $X$ definitely convey information about $X+Y$.
• $I(X+Y;X-Y)$ should be positive because some values of $X-Y$ definitely convey information about $X+Y$. (If $X-Y=0$, we know that $X+Y$ is either $0$ or $2$; if $X-Y=\pm1$, we know that $X+Y=1$.)

For computing $H(X+Y)$ we do have to look at the formula. ($H(X-Y)$ will be the same). $X+Y$ takes on values $0, 1, 2$ with probabilities $\frac14, \frac12, \frac14$, so the surprisal of $X+Y$ takes on values $2, 1, 2$ with probabilities $\frac14, \frac12, \frac14$, and we get $H(X+Y) = 2 \cdot \frac14 + 1 \cdot \frac12 + 2 \cdot \frac14 = 1.5$. So your computation checks out.

Now, we get $$I(X+Y;X) = H(X+Y) + H(X) - H(X+Y,X) = 1.5 + 1 - 2 = 0.5$$ and $$I(X+Y;X-Y) = H(X+Y) + H(X-Y) - H(X+Y,X-Y) = 1.5 + 1.5 - 2 = 1.$$