How to construct an example for the entropy equation: $H(Z)=H(X)+H(Y)$ where $Z=X+Y$

Question

Given $Z=X+Y$ where X and Y are two random variables, under what conditions does $H(Z)=H(X)+H(Y)$?

Notice $Z$ is a function of $(X,Y)$, therefore $H(Z)\leq H(X,Y)$, and $H(X,Y)\leq H(X)+H(Y)-I(X;Y)$. Therefore when $Z$ and $(X,Y)$ is a one-to-one function and $X$ is independent of $Y$.

My question is, how to find an example that satisfy this problem ? I have a simple example that let $X=0$ so that $Z=Y$, which satisfy the conditions. However, can anyone gives a more non-trivial example ? Thanks.

Answer 1

In addition to the nice example @Arash gives. Assume the discrete uniform distribution. If $$X \sim \mathcal U_{[a,b]}(n)$$and$$Y \sim \mathcal U_{[c,d]}(m)$$ Let's define $$Z = X+Y$$ We know that the entropies of $X$ and $Y$ are $$H(X) = \log n$$ and $$H(Y) = \log m$$ therefore $$H(X) + H(Y) = \log(n) + \log(m) = \log(nm)$$ This means that if $Z$ has $nm$ outcomes, then we can assert that $H(Z) = H(X) + H(Y)$. Many examples could be generated here to demonstrate the need of being bijective.

One Example out of MANY

Assume $X \in \lbrace 1,2,3\rbrace$ and $Y \in \lbrace 0,10,20 \rbrace$. It is easy to see that $$Z \in \lbrace 1,2,3,11,12,13,21,22,23 \rbrace$$ with $$P(Z = z) = \frac{1}{9}$$ Is $H(Z) = \ln(3\times 3) = \log 9$ ? Of course, because we $X+Y$ here is bijective.

Let's prove it: $$H(Z) = \sum\limits_{k=1}^9 p_k \log(\frac{1}{p_k}) = \sum\limits_{k=1}^9 \frac{1}{9} \log(9) =9\frac{1}{9} \log(9) = \log(9)$$

Counter example

Now let's take the same distributions (with different outcomes) but show that $H(Z)$ will not reach $\log(9)$ if there is no bijectivity. Let $X \in \lbrace 0,1,2 \rbrace$ and $Y \in \lbrace 3,4,5 \rbrace $. So $$Z \in \lbrace 3,4,5,6,7 \rbrace$$ Therefore, it is clear there is no bijectivity, because $(1,3)$ and $(0,4)$ both give $Z = 4$, hence we can not decode $X,Y$ from $Z$. Computing the entropy $$H(Z) = \frac{1}{9} \log 9 + \frac{2}{9} \log \frac{9}{2} +\frac{3}{9} \log \frac{9}{3} + \frac{2}{9} \log \frac{9}{2} + \frac{1}{9} \log 9 < \log(9)$$

Answer 2

To construct a proper example, as you found out yourself in a very nice fashion, there is a bijection between $Z$ and $(X,Y)$ (I suppose that random variables are discrete of course).

Let $X'\sim \text{Bernoulli}(\frac 12)$ and $Y\sim \text{Bernoulli}(\frac 12)$. Set $X=2X'$ and take $Z=X+Y$. Then $Z$ is uniformly distributed over $\{0,1,2,3\}$ and $X$ and $Y$ are also uniformly distributed over $\{0,2\}$ and $\{0,1\}$. Therefore it is easy to see that $H(Z)=H(X)+H(Y)$.