Decomposing factorized entropy

302 Views Asked by At

I am trying to figure out how the equation for factorized entropy below is derived. The equation for entropy is $H(Q) = -\sum_x Q(x)\log Q(x)$ where $Q$ is a probability distribution. Let $Q(x) = \prod_i Q_i(x_i)$ (a factorized distribution). Then, this is the derivation(see page 4, slide 1) I have seen for factorized entropy:

$$ H(Q)= - \sum_x Q(x)\log Q(x)$$ $$=-\sum_x(\prod_{i\in x} Q(x_i))\log(\prod_{x_i \in x} Q(x_i))$$ $$=-\sum_x(\prod_{i} Q(x_i))\sum_i \log Q(x_i))$$ $$=-\sum_i\sum_{x_i}Q(x_i)\log Q(x_i)$$

I can't figure out how the last line was derived(replacement of $\prod_iQ(x_i)$ by $\sum_{x_i}Q(x_i)$) . What confuses me is that $ab (\log a + \log b) \equiv \log (ab)^{ab}$ is not same as $(a+b)(\log a + \log b) \equiv \log (ab)^{a+b}$. I will appreciate pointers on how the last line is derived.

1

There are 1 best solutions below

0
On

$\text{Line 3 }=-\sum_x(\Pi_jQ_j(x_j))\sum_i\log Q_i(x_i)=-\sum_xQ(x)\sum_i\log Q_i(x_i)=-\sum_i\sum_xQ(x)\log Q_i(x_i)=-\sum_i\sum_{x_i,x_{-i}}Q_{-i}(x_{-i})Q_i(x_i)\log Q_i(x_i)=-\sum_i(\sum_{x_{-i}}Q_{-i}(x_{-i}))(\sum_{x_i}Q_i(x_i)\log Q_i(x_i))=-\sum_i\sum_{x_i}Q_i(x_i)\log Q_i(x_i)=\text{last line}$

$x_{-i}$ means non-$x_i$ terms.

$\sum_{x_i,x_j,x_k}Q_i(x_i)Q_j(x_j)Q_k(x_k)=\sum_{x_i}Q_i(x_i)\sum_{x_j}Q_j(x_j)\sum_{x_k}Q_k(x_k)$, as long as $x_i,x_j,x_k$ are pairwise/mutually independent, which is necessary for $Q$ to be factorized.