Maximum entropy of discrete random variable on natural numbers

143 Views Asked by At

I'm trying to find distribution on natural numbers with maximum entropy, such that its expectation is equals 2 with Lagrange method. My problem is that i can not solve system of equations

$$ L(\lambda_1, \lambda_2, p_1, p_2, ..., p_n, ...) = -\sum p_n\log(p_n) + \lambda_1(\sum np_n - 2) + \lambda_2(\sum p_n - 1)$$

It's something like $$ L'_{p_n} = -\log(p_n) - 1 + \lambda_1n + \lambda_2 $$ $$L'_{\lambda_1} = \sum np_n - 2$$ $$L'_{\lambda_2} = \sum p_n - 1$$

and then

$$p_n = e^{\lambda_1n + \lambda_2 - 1} \\ \sum ne^{\lambda_1n + \lambda_2 - 1} = 2 \\ \sum e^{\lambda_1n + \lambda_2 - 1} = 1$$


1

There are 1 best solutions below

0
On BEST ANSWER

just evaluate the summations to get relations between $\lambda_1$ and $\lambda_2$

$\sum e^{\lambda_1 n + \lambda_2 - 1} = 1\\ e^{\lambda_2 - 1}\sum e^{\lambda_1 n} = 1 \\ e^{\lambda_2 - 1} \frac{e^{\lambda_1}}{1 - e^{\lambda_1}} = 1 \rightarrow (1)$

for other summation, we can use the fact that $\sum_{n = 1}^\infty n*r^n = \frac{r}{(1-r)^2}$, when $|r| < 1$

$\sum ne^{\lambda_1 n + \lambda_2 - 1} = 2\\ e^{\lambda_2 - 1} \sum ne^{\lambda_1 n} = 2\\ e^{\lambda_2 - 1} \frac{e^{\lambda_1}}{(1 - e^{\lambda_1})^2} = 2$

using (1) we get $\frac{1}{1 - e^{\lambda_1}} = 2\\ \frac{1}{2} = 1 - e^{\lambda_1} \\ e^{\lambda_1} = \frac{1}{2} \\ \lambda_1 = -ln(2)$

substituting in (1) we get $e^{\lambda_2 - 1} = 1 \implies \lambda_2 = 1$

so $p_n = \frac{1}{2^n}$ and the entropy is $-\sum \frac{1}{2^n} log(\frac{1}{2^n}) = log(4) \approx 1.38629$