What is the correct equation for conditional relative entropy and why

Question

I was trying to understand the concept of conditional relative entropy.

As in:

$$D(P(X\mid Y) ||Q(X\mid Y))= E [\log\frac{P(X\mid Y)}{Q(X\mid Y)}]$$

I would have thought that its equations would be:

$$D(P(X\mid Y) ||Q(X\mid Y)) = \sum_{x,y} p(x\mid y) \log \frac{p(x\mid y)}{q(x\mid y)}$$

However, the text book that I was reading said the following:

We can think of conditional relative entropy $D(P(X\mid Y) ||Q(X\mid Y))= E [\log\frac{P(X\mid Y)}{Q(X\mid Y)}]$ as a weighted sum of the relative entropies between the conditional distributions given different values of y

$$D(P(X\mid Y) ||Q(X\mid Y)) = \sum_y p(y) D(P(X\mid y) ||Q(X\mid y)) $$

but with a little algebra, the previous interpretation leads me to the following equation:

$$D(P(X|Y) ||Q(X|Y)) = \sum_{x,y} p(x,y) \log \frac{p(x|y)}{q(x|y)}$$

however, it is not the equation that I expected:

$$D(P(X\mid Y) ||Q(X\mid Y)) = \sum_{x,y} p(x\mid y) \log \frac{p(x\mid y)}{q(x\mid y)}$$

Does somebody understand the difference between them and which one is the correct version and why?

Answer 1

For discrete distributions, a "relative entropy" (ordinary or conditional) is by definition an expected value of the logarithm of a ratio of probability mass functions, whereas the expression you consider, viz.

$$\sum_{x,y} p(x\mid y) \log \frac{p(x\mid y)}{q(x\mid y)} $$

is not of the required form, because

$$\sum_{x,y} p(x\mid y) \ne 1. $$

On the other hand,

$$\sum_{x} p(x\mid y) = 1, $$

so the expression

$$\sum_{x} p(x\mid y) \log \frac{p(x\mid y)}{q(x\mid y)} $$

is of the required form, and the quantity so defined is called the "relative entropy" of the conditional distributions given $Y = y$, denoted by $D(P(X\mid y) ||Q(X\mid y))$; furthermore, because this is a function of $y$ only, it has the expected value

$$\sum_y p(y) D(P(X\mid y) ||Q(X\mid y)) = \sum_y p(y) \sum_{x} p(x\mid y) \log \frac{p(x\mid y)}{q(x\mid y)} = \sum_{x,y} p(x, y) \log \frac{p(x\mid y)}{q(x\mid y)} $$

which, by an abuse of terminology and notation, is called the "conditional relative entropy" denoted by $D(P(X\mid Y) ||Q(X\mid Y)$.