KL Divergence: Show that $D(P_{X,Y} \|Q_{X,Y}) = D(P_X\|Q_X) + D(P_{Y |X}\|Q_{Y\|X}|P_X)$

23 Views Asked by At

Using chain rule of KL Divergence, prove that $$D(P_{X,Y} \|Q_{X,Y}) = D(P_X\|Q_X) + D(P_{Y |X}\|Q_{Y\|X}|P_X)$$ where $P_{X,Y}$ and $Q_{X,Y}$ are two joint distributions on $\cX \times \cY$ such that $P_{X,Y} (x,y) = P_X(x) P_{Y|X}(y|x)$ and $Q_{X,Y} (x,y) = Q_X(x) Q_{Y|X}(y|x)$.

Thanks a lot!