I am interested in understanding the following expression:
$$-\frac{1}{M}\sum_{i=1}^M \log \left(\frac{P(x|y=y_i)}{\frac{1}{M}\sum_{j=1}^MP(x|y=y_j)}\right)$$
It seems to be somewhat analogous to entropy as it is minimised when $P(x|y=y_i)$ is constant for all $i$. However, unlike entropy, there does not seem to be any form of symmetry.
My question is if this expression makes sense to be interpreted as something like entropy or variance for a set of conditional probabilities $P(x|y=y_i)$ for $i = 1, \dots , M$? I would also appreciate any references to where I can read more about this expression.
Let $Y \sim Unif \{1,...,M\}$ with probability mass function $p_Y(y) = \frac{1}{M}.$
The conditional distribution is $X\vert Y \sim p_{X\vert Y}(x \vert y)$, such that the joint distribution is $(X,Y) \sim p_{(X,Y)}(x,y)$. From this one can then also obtain the conditional distribution $p_{Y\vert X}(x)$ and the marginal distribution $p_X(x)$.
The above expression can be written as $$\mathbb{E}_Y \left[\ln \frac{p_X(x)}{p_{X \vert Y}(x \vert Y)}\right]$$
Now, using $p_{X \vert Y}(x \vert y) = p_{Y\vert X}(y \vert x) \frac{p_X(x)}{p_Y(y)}$, we can rewrite the above as
$$\mathbb{E}_Y \left[\ln \frac{p_X(x)}{p_{X \vert Y}(x \vert Y)}\right] = \mathbb{E}_Y \left[\ln {p_X(x)} - \ln p_{Y\vert X}(Y \vert x) - \ln p_X(x) + \ln p_Y(Y)\right] = \mathbb{E}_Y \left[ \frac{\ln p_Y(Y)}{\ln p_{Y\vert X}(Y \vert x)}\right].$$
But this is exactly the Kullback-Leibler divergence of the marginal distribution of $Y$ (the uniform distribution) and the distribution when conditioned on $X=x$, that is $D_{KL}\left(p_Y(y) \vert \vert p_{Y\vert X}(Y\vert x)\right).$
And of course, up to a constant, this is the entropy of the conditional distribution.