Problem
$f_i, g_i$ are the probability density functions of the symmetric, unimodal distributions with the common center $c_i$. Assume the following:
- All $g_i$ are the translations of $g_0$ (whose center is $0$). Thus, $h(g_i) = h(g_0), \forall i=1,\cdots,n$.
- $h(f_i) \ge h(g_0)$, $\forall i$
where $h(\xi)$: differential entropy of the random variable following the density function $\xi$. Let's consider two mixture distributions with the shared weights, $\displaystyle f=\sum_{i=1}^n w_i f_i$, $\displaystyle g=\sum_{i=1}^n w_i g_i$, such that $\displaystyle \sum_{i=1}^n w_i = 1$.
Now, I want to prove or disprove my intuition here:
$\displaystyle h(f) \ge h(g)$
Try
I am literally stuck at the very first part: $$ \begin{aligned} h(f) &= h\left(\sum_{i=1}^n w_i f_i\right) \\ &= - \int_{-\infty}^\infty \left(\sum_{i=1}^n w_i f_i (x) \right) \log \left(\sum_{i=1}^n w_i f_i (x) \right) dx \end{aligned} $$
Any suggestions will be welcomed. Many thanks!
Additional remarks
- I encountered this problem when dealing with kernel density estimation (KDE) with kernel $g_i$.
- It would be nonetheless very helpful to know the results when we limit all the distributions $f_i, g_i$ to be Gaussian, which is usually the case in KDE.
Not an answer, but a simple counterxample if the distributions are not restricted to be unimodal.
Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$.
Let $u_{c,w}$ denote the uniform distribution with center $c$ and width $w$ (that is, with support over $[c-w/2,c+w/2]$)
Let $g_0 = u_{0,2}$. Then $g=u_{1,4}$ and $h(g)=\log 4 = 2$ bits.
Let $f_0= \frac{1}{2}(u_{-1,1}+u_{1,1})$, with $f_i$ given by the same translations as $g_i$. Then $h(f_i)=h(g_i)=1$ and
$$f= \frac{1}{4}u_{-1,1}+\frac{1}{2}u_{1,1}+\frac{1}{4}u_{3,1}$$
with $h(f) = h(u_{1,1}) + h_d(\frac{1}{4},\frac{1}{2},\frac{1}{4})= 0 + \frac{3}{2} = 1.5$ bits
It suffices to make a small perturbation to $f_i$ (change its width from $1$ to $1+\epsilon$) to attain $h(f_i)>h(g_i)$ and keep $h(f)<h(g)$
For the unimodal case.
Take $n=2$ , $c_1=0$, $c_2=2$ and $w_i=1/2$, as before, and let $g$ be the same as before.
Let $$f_0= \beta u_{-1,1} + \alpha u_{0,1} + \beta u_{0,1} $$ with $ 0<\alpha <1$ and $\beta = \frac{1-\alpha}{2}<\alpha$. Then $h(f_i)= h_d(\beta,\alpha,\beta)$. And
$$f= \frac{\beta}{2} u_{-1,1} + \frac{\alpha}{2} u_{0,1} + {\beta} u_{1,1} + \frac{\alpha}{2} u_{2,1} + \frac{\beta}{2} u_{3,1}$$
with $h(f)=h_d(\frac{\beta}{2}, \frac{\alpha}{2}, \beta, \frac{\alpha}{2} ,\frac{\beta}{2})$
where $h_d$ denotes the discrete entropy. By choosing $\alpha = \frac{3}{4}$ we get $h(f_i)=1.061278 > 1 = h(g_i)$ but $h(f)=1.936278 < 2 = h(g)$
Of course, $g_0$ and $f_0$ need only an arbitrarily small perturbation to make them strictly unimodal.