Let random variable $X$ be defined over alphabet $X = \{-2, 0, 2\}$.
a) Find the distribution $p(x)$ that maximizes the entropy $H(X)$ while maintaining $E\{|X|\} = \theta$, where $\theta \in [0, 2]$. What is this maximum $H(X)$?
b) Find distribution $q(x)$ that minimizes the divergence $D[q||p]$, with respect to the entropy maximizing $p(x)$ of part a, subject to the constraint $E\{|X|\} = \phi$, where $\phi \in [0, 2]$. What is the entropy of $q(x)$?
My attempt:
Part(a): I used the Lagrangian method and found that $$p_i = \frac{e^{-\lambda|i|}}{\sum_k e^{-\lambda |k|}},$$ such that $i, k\in X$. That is $p_i$ is Gibbs distribution. I am stuck at the point of imposing the average constraint.
Part(b): I started with the Lagrangian method and found that $$q_i = \frac{p_i e^{-\lambda|i|}}{\sum_k p_k e^{-\lambda |k|}},$$ However, I need to impose now two constraints ($\theta, \phi$), which I am not sure how to do.
Any Help!
Thanks in advance.
Well, for part $a)$, you don't even really need to do the multiplier stuff. $|X|$ is supported either on $0$ or $2$. There is only one binary law supported on $\{0,2\}$ with mean $\theta,$ namely $P(|X| = 2) = \theta/2.$ Then to maximise entropy the mass should be spread equally on $+2$ and $-2$, giving you the law $p = (\theta/4, 1-\theta/2, \theta/4)$.
If you prefer the explicit multipliers approach, then you need to work out $\mathbb{E}_P[|X|]$ in terms of $\lambda,$ and solve for the constraint. In other words, you need to solve the equation $$ \sum |i| p_i = \theta. $$ This is straightforard to work out. $$ \sum |i|p_i = \frac{2 e^{-2\lambda} + 0 + 2 e^{-2\lambda}} {e^{-2\lambda} +1 + e^{-2\lambda}} = \theta\iff (4-2\theta)e^{-2\lambda} = \theta \iff e^{-2\lambda} = \frac{\theta }{4-2\theta}.$$
Then $2 e^{-2\lambda} + 1 = \frac{2\theta}{4-2\theta} + 1 = \frac{4}{4-2\theta},$ and so your law works out to $$ P(2) = P(-2) = \frac{e^{-2\lambda}}{2e^{-2\lambda} + 1} = \frac{\theta}{4-2\theta} \cdot \frac{4-2\theta}{4} = \frac\theta 4,$$ et c.
Similarly for part b), you need to work out $\mathbb{E}_q[|X|]$ in terms of $\lambda$, and set this to $\phi$.