How to prove the inequality on relative entropy?

184 Views Asked by At

the definition of relative entropy

Here is the definition of Relative Entropy Now I am only interested in the simplest condition that the index set is finite and discrete, as the naive probability distribution vectors.

Now if the distribution vector $p_3=a p_1+(1-a) p_2$, $q_3=a q_1+(1-a)q_2$ for $0<a<1$ how to prove that $H(p1|q1)<H(p3|q3)<H(p2|q2)$?

1

There are 1 best solutions below

1
On

The right hand side can be proved with the joint convexity of relative entropy, that is,

H(\sum_{i}c_i P_i||\sum_i c_iQ_i)\leq \sum_i c_i H(P_i||Q_i)

the left hand side is not valid as I see. For example, suppose P_1=Q_2=(0.3,0.7),P_2=Q_1={0.7,0.3}, and a=0.5, then the middle term vanishes while the left side is positive.