Bound on entropy of n variables

54 Views Asked by At

Let $X_1,X_2,\dots,X_n$ be arbitrary discrete random variables. Prove that $$H(X_1^n)\leq\frac{1}{n-1}\sum_{i=1}^nH(X_1^{i-1},X_{i+1}^n),$$ where for $j\geq i$, $X_i^j$ denotes the block of random variables $(X_i,\dots,X_j)$ and for $j<i$ $X_i^j$ can just be interpreted as an empty constant zero.

My idea was to show this using induction. The base case $n=2$ is obvious from the chain rule: $$H(X_1,X_2)=H(X_1)+H(X_2|X_1)\leq H(X_1)+H(X_2).$$ Then assuming true for $n=k$, the idea was to use the chain rule and hypothesis again: $$H(X_1^{k+1})=H(X_1^k)+H(X_{k+1}|X_1^k)\leq \frac{1}{k-1}\sum_{i=1}^kH(X_1^{i-1},X_{i+1}^k)+H(X_{k+1}|X_1^k).$$ But I'm unsure how to proceed from here because of the fraction making it difficult to merge the terms in a meaningful way. Any advice would be greatly appreciated!

1

There are 1 best solutions below

0
On BEST ANSWER

This fact is often referred to as Han's Inequality. It's hard to use induction to prove this since like you've noticed it gets a bit messy from the $1/(k-1)$. Instead you can use the inequality, valid for all $i=1,...,n$

\begin{align} H(X_1^n) &= H(X_1^{i-1}, X_{i+1}^n) + H(X_i|X_1^{i-1}, X_{i+1}^n) \\ &\leq H(X_1^{i-1}, X_{i+1}^n) + H(X_i|X_1^{i-1}) \end{align} where the first line is the chain rule, and the second line is that conditioning reduces entropy. To finish up, see what happens when you sum both sides over $i=1,...,n$. As a hint, think about the value of this sum $$\sum_{i=1}^n H(X_i|X_1^{i-1}).$$