Maximize $J[f] = \int_\mathbb{R} f(x)\log f(x)\,dx$ over smooth surjections $f : \mathbb{R}\to (0, \alpha)$ subject to $\int_\mathbb{R} f(x)\,dx = 1$.

150 Views Asked by At

Maximize $J[f] = \int_\mathbb{R} f(x)\log f(x)\,dx$ over smooth surjections $f : \mathbb{R}\to (0, \alpha)$, where $\alpha$ is a real number, subject to $\int_\mathbb{R} f(x)\,dx = 1$.

I have no idea how to do this. I came across this while I was pondering some differential entropy problems.

1

There are 1 best solutions below

2
On BEST ANSWER

The smoothness and surjectivity requirements don't affect the supremum; when dealing with integral expressions like this, we can smoothen things out at arbitrarily small cost in integral terms. The only thing the smoothness constraint can do is prevent the supremum from being attained.

So, let's maximize $\int f\log f$ over functions $f:\mathbb{R}\to [0,\alpha ]$ subject to $\int f=1$. Since $\log$ is increasing, $$\int f\log f\le \int f \log \alpha = \log \alpha$$ Equality is attained by the function $f=\alpha \chi_{[0,1/\alpha]}$, so this is indeed the supremum.

A smooth approximation to $\alpha \chi_{[0,1/\alpha]}$ can be obtained by replacing vertical walls with steep, but smooth ramps. Also, the values can be made to lie strictly between $0$ and $\alpha$ by adding a tiny bit of Gaussian, and rescaling a bit.

Intuitive explanation: you are minimizing the entropy of a distribution with pdf bounded from above. It wants to be as close to deterministic as possible, i.e., sharply localized. So the maximum is attained by putting all mass somewhere with largest allowable density.