Shannon Entropy over sum of random variables

Question

For a project I am working with the JS-Divergence, where the divergence between random variables $P$ and $Q$ is given by $$JSD = H(M) - \frac{1}{2} (H(P) + H(Q)),$$ with $M = \frac{P + Q}{2}$, and $H$ denotes the Shannon entropy over random variable $X$ with possible outcomes $x_1, ..., x_n$, i.e. $$H(P) = - \sum_{i=1}^n P(x_i) \log P(x_i).$$ Considering the first term of JSD, we have that $$H(M) = H(\frac{1}{2}P + \frac{1}{2}Q),$$ but how would I express this in terms of Shannon entropy? Is it the case that $$H(\frac{1}{2}P + \frac{1}{2}Q) = -\sum \frac{1}{2}(P(x_i)+Q(x_i)) \log \frac{1}{2}(P(x_i)+Q(x_i)),$$ and if not, how else can I simplify $H(M)$? I would ideally rewrite the total expression in terms of the form $A \log B$, for random variables $A, B$. Any advise? Thanks in advance!

Answer 1

Note that merely rescaling the values of a discrete random variable does not change its entropy, which merely depends on the probabilities. Accordingly:

$H(\frac{1}{2}P+\frac{1}{2}Q) = H(P+Q)$

It is not immediately clear what we can do about the right-hand side of the above equation. However, we might note that $P+Q$ clearly cannot contain more information than $PQ$ (as the former is a deterministic function of the latter), and we know that:

$H(PQ) = H(P) + H(Q) - I(P;Q)$

Accordingly, we have:

$H(P+Q) \leq H(P)+H(Q)$

Further analysis will require tackling both the $I(P;Q)$ term and any redundancies in the mapping from $PQ$ to $P+Q$.