I have a question about optimizing the following quantity over function form .
Given unknown function $f(\theta)$ such that $f(\theta)\geqslant 0$ and $\int f(\theta)d\theta\leq \infty$. And function $p(\theta)$ is unknown, and maybe we have one more conditon that $\int p(\theta)d\theta=1$.
The goal is to optimize the following integral over function form $p(\theta)$,
$ \max_{p(\theta)} \int p(\theta)\log \frac{f(\theta)}{p(\theta)}d\theta$.
Many notes say that based on elementary exercise in calculus of variations, one can show that the function $p(\theta)$ ,which is proportional to function $f(\theta)$, maximizes the integral, i.e. $p(\theta)\propto f(\theta)$ .
But I have no idea how to do it. Could anyone show me how to maximize the integral using the technique of calculus of variation ? Besides this quantity is very important in information theory. thank you
Without going into too much detail, an extremum w.r.t. $p$ happens as a solution to the Euler-Lagrange equation, which for this functional happens when $p=\frac{1}{e}f$.
EDIT: A "little" more detail: in the first case, the Lagrangian is given by $L=L(x,p,p')=p(x)\log\frac{f(x)}{p(x)}=p\left(\log f-\log p\right)$, and since $L$ doesn't depend explicitly on $p'$, the Euler-Lagrange equation is $$0=\frac{dL}{dp}-\frac{d}{dx}\frac{dL}{dp'}=(\log f-\log p)+p\left(-\frac{1}{p}\right)=\log f-\log p-1=\log\frac{f}{e}-\log p,$$ giving the solution mentioned above.
If then the Lagrangian $L=p(x)\log\frac{f(p(x))}{p(x)}$, the EL-equation is a little more complicated: $$0=(\log f(p)-\log p)+p\left(\frac{f'(p)}{f(p)}-\frac{1}{p}\right)$$ $$\Longrightarrow \frac{p}{f(p)}=\exp\left(\frac{p}{f(p)}f'(p)\right),$$ for which I cannot imagine a way to solve for $p$ in general, so this situation is not the same.