Anyone saw this interesting function before?

Question

Say $\theta\in\Re^n$ and $\theta_i\in(0,1)$ for all $i$. Define

$$ f(\theta) = \frac{1}{n}\sum_i^n\{(1-\theta_i)\log(1-\theta_i)+\theta_i\log\theta_i\} $$

It is easy to see the minimizer of $f(\theta)$ is $(\frac{1}{2},\ldots,\frac{1}{2})^T\in\Re^n$. Also, it is interesting to note that the Hessian of this function is always diagonal matrix.

What can we say about its sublevel set, i.e., $\mathcal{S}_{\alpha}=\{\theta:f(\theta)\leq\alpha\}$?

Is there any analytic expression for $\mathcal{S}_{\alpha}$?

Answer 1

The function is separable in that you can write it as $f(\theta) = \sum_i \phi(\theta_i)$, where $\phi$ is the convex function (on $(0,1)$) $\phi(x) = x \ln x + (1-x) \ln (1-x)$. Any function of this form will have a diagonal Hessian. The sum of convex functions is convex, hence $f$ is convex, and the level sets will, of course, be convex.

I doubt that these is a nice analytical expression for the level set.

Answer 2

This is simply the average negative entropy of $n$ independent Bernoulli distributions with means $\{ \theta_i \}_i$. Entropy $H(P) = -\sum_x P(x) log(P(x))$ is a concave function of $P$ and achieves its maximum at uniform distribution.

In this case since distributions are independent, the entropy is decomposed to sum of individual entropies and naturally all derivatives are decomposable as well. I don't think there is any simple analytical form for $\mathcal{S}_\alpha$.