Computing entropy from joint probability table

Question

I'm teaching myself aspects of information theory. I found a book's chapter here: http://www.eecs.harvard.edu/cs286r/courses/fall10/papers/Chapter2.pdf. After reading this chapter a few times over, I still don't understand the following examples answers:

In particular, what does (1/8, 3/4) in H(1/8, 7/8) mean? Why is H(X|Y = 2) = 1? Also, where does the 1/4 coefficient in 1/4 H(X|Y = 2) come from?

EDIT:

P(Y = 1) = P(X = 1, Y = 1) + P(X = 2, Y = 1) = 0 + (3/4) = (3/4) = P(Y = 1)


P(Y = 2) = P(X = 1, Y = 2) + P(X = 2, Y = 2) = (1/8) + (1/8) = (1/4) = P(Y = 2)

Therefore,

H(X | Y = 1) = H( P(X = 1 | Y = 1), P(X = 2 | Y = 1)) = H(0, 1-(3/4)) = H(0, (1/4))


H(0, (1/4)) = (0 * log2 (0)) + ((1/4) * log2 (4))

but log2 (0) is undefined?? So, what I'm not understanding when computing H(X | Y = 1)?

Answer 1

• The notation $H(p,1-p)$ means entropy of a binary-valued random variable taking its two values with probability $p$ and $1-p$, respectively (the actual values are irrelevant).
• The notation $H(X|Y=2)$ means entropy of the random variable $X$ conditioned on the event $Y=2$. Note that, conditioned on $Y=2$, $X$ is a binary-valued random variable taking its two values with probabilities $\mathbb{P}(X=1|Y=2)=\mathbb{P}(X=1,Y=2)/\mathbb{P}(Y=2)$ and $\mathbb{P}(X=2|Y=2)=\mathbb{P}(X=2,Y=2)/\mathbb{P}(Y=2)$, respectively. It follows that corresponding entropy equals $H(\mathbb{P}(X=1|Y=2), \mathbb{P}(X=2|Y=2))$ (using the notation mentioned above). You can compute the required probabilities from the joint distribution given in the table and verify the result.

• By definition,

  $$ \begin{align} H(X|Y) &= \sum_y \mathbb{P}(Y=y) H(X|Y=y)\\ & =\mathbb{P}(Y=1) H(X|Y=1) + \mathbb{P}(Y=2) H(X|Y=2) \end{align} $$