Generalization of the Shannon entropy that looks at the $n$ most probable values

58 Views Asked by At

The Shannon Entropy of a random variable $X$ is given as:

$$ H(X) = \sum_{x \in X} P(x) \log P(x) $$

where the sum is over all possible outcomes of $X$. This can sometimes be difficult to compute, such as if $X$ has a very large space of possibilities. In those situations the min-entropy is a related quantity often much easier to compute:

$$ H_\infty(X) = -\log\left(\max_{x \in X} P(x) \right) $$

To compute this we need only locate the most probable outcome in $X$, which is often much easier than doing a sum on all possible outcomes. Both the Shannon entropy and the Rényi entropy these are special instances of the Rényi Entropy and there are theorems relating the various Rényi entropies to one another.

My question is this: the thing that makes the min-entropy so useful is that we need only look at one probability. This suggests a family of generalized entropies that look at, for instance, the top $n$ probabilities, with $n=1$ being the min-entropy and $n=|X|$ being the Shannon entropy. This would be very useful, as for instance, locating the top two most probable outcomes is often just as easy as locating the most probable one. Playing around with this seems to yield some useful ideas, but I wanted to ask: does something like this already exist? Is there any literature on this, and a name for this, or some standardized way to do it? As mentioned, this would be different from the Rényi entropy, which would interpolate things a different way.