Conditional joint information of two random variables $X,Y$ given $Z$

90 Views Asked by At

For 3 random variables I am trying to prove the following:

\begin{eqnarray*} I(X;Y|Z)&\triangleq& H(X|Z)-H(X|Y,Z)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (1) \\&=&E_{p(x,y,z)}\bigg[log_2\frac{p(X,Y|Z)}{p(X|Z)\cdot p(Y|Z)}\bigg]\ \ \ \ \ (2) \end{eqnarray*}

Starting from the first equality I get: \begin{eqnarray*} (1)\Rightarrow I(X;Y|Z)&\triangleq& H(X|Z)-[H(X,Y|Z)-H(Y|Z)] \\&=&H(X|Z)-H(X,Y|Z)+H(Y|Z) \\&=&\sum\limits_{z}\Big\{p(z)\cdot (H(X|Z=z)-H(X,Y|Z=z)+H(Y|Z=z))\Big\} \\&=&\sum\limits_{z}\Big\{p(z)\Big(-\sum\limits_{x}p(x|z)log_2\{p(x|z)\}+\sum\limits_{x,y}p(x,y|z)log_2\{p(x,y|z)\} \\&&-\sum\limits_{y}p(y|z)log_2\{p(y|z)\}\Big)\Big\} \\&=&\sum\limits_{z}\Big\{-\sum\limits_{x}p(z)p(x|z)log_2\{p(x|z)\}+\sum\limits_{x}\sum\limits_{y}p(z)p(x,y|z)log_2\{p(x,y|z)\} \\&&-\sum\limits_{y}p(z)p(y|z)log_2\{p(y|z)\}\Big\} \\&=&\sum\limits_{z}\Big\{-\sum\limits_{x}p(x,z)log_2\{p(x|z)\}+\sum\limits_{x}\sum\limits_{y}p(x,y,z)log_2\{p(x,y|z)\} \\&&-\sum\limits_{y}p(y,z)log_2\{p(y|z)\}\Big\} \\&=&\sum\limits_{z}\Big\{-\sum\limits_{x}\sum\limits_{y}p(x,z|y)p(y)log_2\{p(x|z)\}+\sum\limits_{x}\sum\limits_{y}p(x,y,z)log_2\{p(x,y|z)\} \\&&-\sum\limits_{y}\sum\limits_{x}p(y,z|x)p(x)log_2\{p(y|z)\}\Big\} \\&=&\sum\limits_{x}\sum\limits_{y}\sum\limits_{z}\Big\{p(x,y,z)log_2\frac{p(x,y|z)}{p(x|z)p(y|z)}\Big\} \\&=&E_{p(x,y,z)}\bigg[log_2\frac{p(X,Y|Z)}{p(X|Z)\cdot p(Y|Z)}\bigg] \end{eqnarray*}

Note that the symbol $\triangleq$ has been used on equalities that generally hold as theoreams or can be easily proven.

1

There are 1 best solutions below

5
On BEST ANSWER

The second equality $ I(X;Y|Z) = -H(X,Y|Z)$ should make you pause - the left side is positive and the right side is negative, something has already gone wrong.

In the previous line, you replaced:

$$H(X|Y,Z)=H(X,Y|Z)-H(X|Z)$$

which is wrong. The correct equality (recall the chain rule: $H(X,Y)=H(X|Y) +H(Y)$ or $H(X|Y) = H(X,Y)-H(Y)$ and condition everything on $Z$) is

$$H(X|Y,Z)=H(X,Y|Z)-H(Y|Z)$$


Update: Regarding the marginalization, take for example the first term

$$ \sum_z p(z) \sum_x p(x|z) (\cdot)=\sum_z \sum_x p(x,z) (\cdot) =\sum_x \sum_y \sum_z p(x,y,z) (\cdot) = E_{p(x,y,z)}[(\cdot)]$$

Update 2: Regarding your doubt in the comment. Yes, that's rather trivial. In detail:

$$ p_X(x)=\sum_y p_{X,Y}(x,y) \implies g(x) \, p_X(x) = g(x) \, \left(\sum_y p_{X,Y}(x,y)\right) = \sum_y g(x) \, p_{X,Y}(x,y) $$