Entropy calculation of splitted experiment

43 Views Asked by At

Let's say, that we have a set of probabilities $X = \{0.4, 0.3, 0.2, 0.1\}$, each for some event of an experiment. The entropy for this set is equal to $H(X) = 1.2798$. Now, if we divide $X$ into two separate ones $X_a = \{0.4, 0.3\}$ and $X_b = \{0.2, 0.1\}$, and normalize them, we get $\tilde{X}_a = \{4/7, 3/7\}$ and $\tilde{X}_b = \{2/3, 1/3\}$. For these two sets we get $H(\tilde{X}_a) = 0.6829$ and $H(\tilde{X}_b) = 0.6365$, respectively. Interpretation of this procedure is not important at this point, just the consecutive steps involved are relevant.

My question is, can we get the original entropy $H(X)$, knowing only $\tilde{X}_a$ and $\tilde{X}_b$? My guess is, that we can only get the upper bound in the form

$$ H(X) \leq H(\tilde{X}_a) + H(\tilde{X}_b), $$

but not the exact value of $H(X)$.

1

There are 1 best solutions below

0
On BEST ANSWER

Assume that you have two random variables $X_a$ and $X_b$. You can define a third variable $X$ by choosing $X_a$ with probability $p$ or $X_b$ with probability $1-p$. In general, this is known as a mixture. The resulting distribution is the weighted average of the component distributions.

In your case, you have a two-component mixture, with non-overlapping distributions. And with $p=0.4+0.3=0.7$, i.e., the normalization factor.

It's easy to prove that the resulting entropy is the weighted average of the entropies plus the entropy of the mixing itself. That is

$$H(X) = p H(X_a) + (1-p) H(X_b) + h(p) \tag 1$$

where $h()$ is the binary entropy function.

Hence yor conjecture is wrong (counterxample: $H(X_a)=H(X_b)=0.1$ , $p=1/2$)

We can, however, notice that $(1)$ is convex on $p$, and hence its maximum is attained at its critical point. This gives a tight bound (in nats):

$$H(X) \le \log( \exp{H(X_a)} +\exp{H(X_b)}) = LSE(H(X_a),H(X_b))$$

where LSE = LogSumExp.