What modification is this of the notion of Renyi divergence?

98 Views Asked by At

Given two probability distributions $P$ and $Q$ over the same outcome and event space (assume finite if needed) one defines their Renyi divergence as $D_\alpha (P \vert \vert Q) = \frac{1}{\alpha -1} \log \left ( \sum _{x \in \Omega} P^{\alpha} (x) Q^{1-\alpha}(x) \right ) $

Now I see the following equations related to this which I am totally unable to parse,

  • $D(P_{Y \vert X} \vert \vert Q_{Y \vert X } \vert P_X ) = D(P_{Y\vert X} P_X \vert \vert Q_{Y \vert X} P_X )$

  • $H(X) - H(X \vert Y) = D(P_{Y \vert X} \vert \vert P_Y \vert P_X ) = D( P_{Y \vert X} P_X \vert \vert P_Y \times P_X)$

  • $D(P_{Y \vert X} \vert \vert Q_Y \vert P_X ) = {\mathbb E} [ D (P_{Y \vert X} (.\vert X ) \vert \vert Q_Y)]$

Can someone help understand these identities?


I am unable to locate any definition for any of these $4$ things,

$D(P_{Y \vert X} \vert \vert Q_{Y \vert X } \vert P_X )$

$D(P_{Y\vert X} P_X \vert \vert Q_{Y \vert X} P_X )$

$D (P_{Y \vert X} (.\vert X ) \vert \vert Q_Y)$

$D( P_{Y \vert X} P_X \vert \vert P_Y \times P_X)$

1

There are 1 best solutions below

8
On

The Renyi divergence is a family of quantities of which the relative entropy $D(P\|Q)$ is a special case when $\alpha = 1$.

The identities you listed involve the entropy and relative entropy of random variables $X$ and $Y$. For example, your first identity is:

\begin{align} D(P_{Y \vert X} \vert \vert Q_{Y \vert X } \vert P_X ) &= \sum_x P_X(x) \sum_y P_{Y|X}(y|x)\log \frac{P_{Y|X}(y|x)}{Q_{Y|X}(y|x)} \\ &= \sum_{x,y} P_{Y|X}(y|x) P_X(x)\log \frac{P_{Y|X}(y|x)P_X(x)}{Q_{Y|X}(y|x)P_X(x)} \\ &= D(P_{Y\vert X} P_X \vert \vert Q_{Y \vert X} P_X ) \end{align}


Update:

$Q_{Y|X}$ and $P_{Y|X}$ are conditional probability distributions. $P_X$ is a probability distribution. Everything I wrote is just definition.

The identities you listed don't have anything to do with Renyi divergence, so they don't follow from its definition.

Entropy $H$ and relative entropy $D$ are functions of probability distributions. $$H(P) = -\sum_x P(x)\log P(x)$$ $$D(P\|Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)}$$