Joint entropy of 2 independent random variables

Question

Say we have two independent random variables $X$ and $Y$. What is their joint entropy $H(X,Y)$?

I worked this out, but I am not sure if the result I reached is correct.

The definitions of entropy that I used are:

$ \begin{align} H(X) = - \sum_{x \in D(X)} P(x)\log_2P(x) \\ H(X,Y) = - \sum_{x \in D(X)} \sum_{y \in D(Y)} P(x,y)\log_2P(x,y) \end{align} $

I started from the definition of $H(X,Y)$ and rewrote it using the assumption that $X$ and $Y$ are independent.

$ \begin{align*} H(X,Y) = - \sum_{x \in D(X)} \sum_{y \in D(Y)} P(x)P(y)\log_2(P(x)P(y)) \end{align*} $

From here I used algebra rules. However due to the double summation, I am not confident in my result.

$ \begin{gather} H(X,Y) = - \sum_{x \in D(X)} P(x) \sum_{y \in D(Y)} P(y)(\log_2P(x)+ \log_2P(y)) \tag{1}\\ = - \sum_{x \in D(X)} P(x) \sum_{y \in D(Y)} (P(y)\log_2P(x)+ P(y)\log_2P(y)) \tag{2}\\ = - \sum_{x \in D(X)} P(x)\log_2P(x) \sum_{y \in D(Y)} (P(y)+ P(y)\frac{\log_2P(y)}{\log_2P(x)}) \tag{3}\\ = H(X) \sum_{y \in D(Y)} (P(y) + P(y)\frac{\log_2P(y)}{\log_2P(x)}) \tag{4}\\ = H(X) (\sum_{y \in D(Y)} P(y) + \sum_{y \in D(Y)} P(y)\frac{\log_2P(y)}{\log_2P(x)}) \tag{5}\\ = H(X) (\sum_{y \in D(Y)} P(y) - \frac{H(Y)}{\log_2P(x)}) \tag{6}\\ = H(X) (1 - \frac{H(Y)}{\log_2P(x)}) \tag{7}\\ = H(X) - \frac{H(Y)H(X)}{\log_2P(x)} \tag{8} \\ = H(X) - H(Y)\sum_{x \in D(X)}P(x)\frac{\log_2P(x)}{\log_2P(x)} \tag{9}\\ = H(X) - H(Y)\sum_{x \in D(X)}P(x) \tag{10} \\ = \boxed{H(X) - H(Y)} \tag{11} \end{gather} $

Thanks.

EDIT:

New method: expand summations

Starting from the rewritten expression of $H(X,Y)$, expand the inner sum (assume there are $n$ elements in $Y$).

$$\begin{align}& H(X,Y) \\[2ex]&= - \sum_{x} \sum_{y} P(x)P(y)(\log P(x) + \log P(y)) \\[2ex]&= - \sum_{x} \sum_{y} (P(x)P(y)\log P(x) + P(x)P(y)\log P(y)) \\[2ex]&= - \sum_{x} P(x) (P(y_1)\log P(x) + P(y_1)\log P(y_1) + \ldots + P(y_n)\log P(x) + P(y_n)\log P(y_n)) \\[2ex]&= - \sum_{x} P(x) (P(y_1)\log P(x) + \ldots + P(y_n)\log P(x) + P(y_1)\log P(y_1) + \ldots + P(y_n)\log P(y_n) ) \\[2ex]&= - \sum_{x} P(x) (\log P(x)(P(y_1) + \ldots + P(y_n)) + \sum_{y} P(y) \log P(y) ) \\[2ex]&= - \sum_{x} P(x) (\log P(x) - H(Y)) \\[2ex]&= - \sum_{x} (P(x)\log P(x) - P(x)H(Y)) \\[2ex]&= - (P(x_1)\log P(x_1) - P(x_1)H(Y) + \ldots + P(x_n)\log P(x_n) - P(x_n)H(Y)) \\[2ex]&= - (P(x_1)\log P(x_1) + \ldots + P(x_n)\log P(x_n) - P(x_1)H(Y) - \ldots - P(x_n)H(Y)) \\[2ex]&= - (\sum_{x}P(x)\log P(x) - H(Y)(P(x_1) + \ldots P(x_n))) \\[2ex]&= \boxed{H(X) + H(Y)} \end{align}$$

Answer 1

When you rewrote $-\sum P(x) \log_2 P(x)$ as $H(x)$ on the left hand side, you neglected the dependency $1/\log_2 P(x)$ on the right hand side on the summation.

Answer 2

Yes, indeed. Or with a little cleanup:

$$\begin{align}&\qquad \mathsf H(X,Y) \\[2ex]&= -\sum_{x,y} \mathsf P_{X,Y}(x,y)\log_2\mathsf P_{X,Y}(x,y) \\[2ex]&= -\sum_{x,y}\mathsf P_X(x)\mathsf P_Y(y)\log_2\left(\mathsf P_X(x)\mathsf P_Y(y)\right) \\[2ex]&= -\sum_{x,y}\mathsf P_X(x)\mathsf P_Y(y)\log_2\mathsf P_X(x) -\sum_{x,y}\mathsf P_X(x)\mathsf P_Y(y)\log_2\mathsf P_Y(y) \\[2ex]&={- \left(\sum_x\mathsf P_X(x)\log_2\mathsf P_X(x)\right)\left(\sum_y\mathsf P_Y(y)\right)-\left(\sum_x\mathsf P_X(x)\right)\left(\sum_y\mathsf P_Y(y)\log_2\mathsf P_Y(y)\right)} \\[2ex]&=\mathsf H(X)+\mathsf H(Y) \end{align}$$