Maximize entropy under Kulback-Leibler divergence

63 Views Asked by At

In this question, it is to solve

\begin{align} \max_p &-\int dy\,p(y)\ln p(y) \\ 0&=1-\int_{-\infty}^{\infty}p(y)dy\\ 0&=1-\int_{-\infty}^{\infty}y^2p(y)dy\\ 0&=c+\int_{-\infty}^{\infty}\frac{e^{-y^2/2}}{\sqrt{2\pi}}\ln p(y)\,dy \end{align} I can certainly apply the calculus of variation. However I would like to directly apply Jensen's inequality.

Here is my attempt at the solution. But I am not sure about the validity of the last inequality. Would someone please correct me? \begin{align} &\int dy\, p(y)\ln \frac1{p(y)} \\ =& \int dy\,p\ln\frac{\exp\big(-\lambda_1y^2-\frac{\lambda_2}{\sqrt{2\pi}}e^{-\frac{y^2}2}\frac{\ln p}{p}\big)}{p}+\lambda_1 \int dy\, y^2 p +\lambda_2 \int dy\,e^{-\frac{y^2}2}\ln p \quad \text{where} p:=p(y) \\ \le&\ \ln\int dy \exp\Big(-\lambda_1y^2-\frac{\lambda_2}{\sqrt{2\pi}}e^{-\frac{y^2}2}\frac{\ln p}{p}\Big)+\lambda_1 \int dy\, y^2 p +\frac{\lambda_2}{\sqrt{2\pi}} \int dy\,e^{-\frac{y^2}2}\ln p \end{align} where the last inequality comes from Jensen's inequality applied to $\ln$ which is convex. The equality is reached when and only when $p=p_1$ such that $\exp\big(-\lambda_1y^2-\frac{\lambda_2}{\sqrt{2\pi}}e^{-\frac{y^2}2}\frac{\ln p_1}{p_1}\big)=z(\lambda_1,\lambda_2)p_1$ for some positive function $z(\lambda_1,\lambda_2)$ so that \begin{align} \int dy\,p_1(y)&=1, \\ \int dy\,y^2p_1(y)&=1, \\ \int dy\,\frac1{\sqrt{2\pi}}e^{-y^2/2}p_1(y)&=-c. \end{align} We have $$\int dy\, p_1\ln\frac1p_1=\ln z(\lambda_1,\lambda_2)+\lambda_1-\lambda_2c.$$ However, I am not sure if $$\int dy\, p\ln\frac1p\le\int dy\, p_1\ln\frac1p_1,\quad\forall \text{ probability measure }p.$$