Maximum Entropy Distribution When Mean and Variance are Not Fixed with Positive Support

Question

I know when the mean and variance of $\ln x$ are both fixed, then the maximum entropy probability distribution is lognormal. When the mean of a random variable is fixed the MEPD is the exponential distribution. My question is, what is the MEPD in the continuous case when neither the mean or variance are fixed with support on $[0, \infty)$?

Answer 1

• Discrete case

In the discrete case you need to consider the functional

$$H[p]=-\sum_{i=1}^n p_i \ln(p_i)+\lambda(\sum_{i=1}^n p_i-1)$$

as we consider a single constraint.

Setting $\frac{\partial H[p]}{\partial p_i}=0$ for all $i=1,\dots,n$ we arrive at

$$-\ln(p_i)-1+\lambda=0\Leftrightarrow p_i=e^{\lambda-1}.$$

Imposing $\sum_{i=1}^n p_i-1=0$, one gets

$\lambda=1-\ln(n)$, or $p_i=e^{1-\ln(n)-1}=\frac{1}{n}$.

In summary, the wished distribution is the uniform probability distribution.

• Continuous case

The continuous case needs more care, due to non trivial integration range. We want to maximize the functional

$$H[p]=-\int_{0}^{\infty}p(x)\ln(p(x))dx+\lambda(\int_{0}^{\infty}p(x)dx-1), $$

where $p$ has support $[0,\infty]$ and $p(0)=p(\infty)=0$. We apply the calculus of variations by considering any distribution $\phi$ s.t. $p(0)=p(\infty)=\phi(0)=\phi(\infty)$. We compute the variation

$$\frac{\delta H}{\delta\phi}|_{p}=\lim_{\epsilon\rightarrow 0} \frac{H[p+\epsilon\phi]-H[p]}{\epsilon}=\lim_{\epsilon\rightarrow 0}\frac{1}{\epsilon}\left[\int_{0}^{\infty}\left(F(p+\epsilon\phi,x)-F(p,x)\right)dx+ \lambda(\int_{0}^{\infty}\epsilon\phi dx)\right],$$

where $F(p,x)=-p(x)\ln(p(x))$ and $F(p+\epsilon\phi,x)=-(p(x)+\epsilon\phi)\ln(p(x)+\epsilon\phi)$.

Using

$$F(p+\epsilon\phi,x)-F(p,x)=\epsilon\phi\frac{\partial F}{\partial p}(p,x)+O(\epsilon^2)$$

we have

$$\frac{dH}{d\phi}|_{p}=\int_{0}^{\infty}\left(\frac{\partial F}{\partial p}(p,x)+\lambda\right)\phi dx$$

where $\frac{\partial F}{\partial p}(p,x)=-\ln(p(x))-1$. In summary

$$-\ln(p(x))-1+\lambda=0 $$

or $p(x)=e^{\lambda-1}$, with $\int_0^{\infty}e^{\lambda-1}dx=1,$ which is not possible.

Roughly speaking, the absence of additional constraints like the fixed mean one

$$\int_{0}^{\infty} xp(x)dx=\mu$$

does not allow to arrive at "more interesting" differential equations for $p(x)$. Note that the $F$ does not depend on $p'(x)$: this leads to the simplified Euler Lagrange equation

$$\frac{\partial F}{\partial p}(p,x)+\lambda=0.$$