How to properly integrate an entropy formula?

Question

Entropy, denoted as $H$, is

$$ H = - \int_a^b f(x)\log(f(x))\mathsf dx$$

where $f$ is given by the classic:

$$ f(x) = \frac1{\sigma\sqrt{2\pi}}e^{-\frac12\left(\frac{x-\mu}\sigma\right)^2}.$$

Here is what I have tried so far to analytically solve this integral:

1. Integration by parts (too messy to write all out here), but quickly realized I needed a fancy substitution, thus leading me to:

2. Convert from $x$-space (where the range is $a$ to $b$) to t-space (where the range is $\alpha$=$\frac{a-\mu}{\theta}$ to $\beta$=$\frac{b-\mu}{\theta}$) and, in general, $t$=$\frac{x-\mu}{\theta}$.

I think I'm on the right track (the $\theta$ in the denominator nicely cancels using the substitution approach) but I got stuck going from x-space to t-space and back again (i.e., I'm in xt-space purgatory and I want out!). Can anyone help me make more headway into #2?

Answer 1

Hand calculation is not that painful. Fortunately Wikipedia gives the result we are aiming for.

The density of the truncated normal over the interval $[a,b]$ is $$f(x)=\frac1{\sigma Z}\phi\left({x-\mu\over\sigma}\right)$$ where $\phi$ is the standard normal density and $Z$ is the normalizing constant $$Z:=\int_\alpha^\beta\phi(t)\,dt\;, $$ where the limits of integration are $\alpha:=(a-\mu)/\sigma$ and $\beta:=(b-\mu)/\sigma$. Using your substitution $t:=(x-\mu)/\sigma$, the entropy is $$ \begin{align} H&:=-\int_a^bf(x)\log f(x)\,dx = -\frac1Z\int_\alpha^\beta\phi(t)\log\left(\phi(t)/\sigma Z\right)\,dt\\ &= -\frac1Z\int_\alpha^\beta\phi(t)\log\phi(t)\,dt +\frac1Z\int_\alpha^\beta\phi(t)\log(\sigma Z)\,dt\\ &= -\frac AZ + B, \end{align} $$ where we calculate $B=\log(\sigma Z)$, and $$ \begin{align} A&=\int_\alpha^\beta \phi(t)\log\phi(t)\,dt\\ &=\int_\alpha^\beta \phi(t)\left(-{t^2\over2}-\log\sqrt{2\pi}\right)\,dt\\ &=\int_\alpha^\beta -{t^2\over2}\phi(t)\,dt - \int_\alpha^\beta\phi(t)\log\sqrt{2\pi}\,dt\\ &=C-Z\log\sqrt{2\pi}\;, \end{align} $$ and, using integration by parts, $$\begin{align} C=\int_\alpha^\beta -{t^2\over2}\phi(t)\,dt &=\int_\alpha^\beta\frac t2\,d\phi(t)\\ &=\frac t2\phi(t)\left.\right]_\alpha^\beta-\frac12\int_\alpha^\beta\phi(t)\,dt\\ &={\beta\phi(\beta)-\alpha\phi(\alpha)\over2}-\frac Z2\;. \end{align} $$ Putting it all together: $$ \begin{align} H&=-\frac1Z \left({\beta\phi(\beta)-\alpha\phi(\alpha)\over2}-\frac Z2-Z\log\sqrt{2\pi}\right) +\log(\sigma Z)\\ &={\alpha\phi(\alpha)-\beta\phi(\beta)\over2Z}+\log\left(\sqrt{2\pi e}\sigma Z\right) \end{align} $$

Answer 2

Why try to perform this by hand, rather than symbolically by computer?

$\int\limits_a^b N(\mu, \sigma; x) {\rm ln}(N(\mu, \sigma; x)) dx =$

$ \frac{\sqrt{\frac{2}{\pi }} \left(b e^{-\frac{b^2}{2}}-a e^{-\frac{a^2}{2}}\right)+(1+\log (2)+\log (\pi )) \text{erf}\left(\frac{a}{\sqrt{2}}\right)-(1+\log (2)+\log (\pi )) \text{erf}\left(\frac{b}{\sqrt{2}}\right)}{2 \left(\text{erf}\left(\frac{b}{\sqrt{2}}\right)-\text{erf}\left(\frac{a}{\sqrt{2}}\right)\right)}$

Here's the Mathemtica code:

FullSimplify@(Assuming[a < b && {a, b} \[Element] Reals, 
   Integrate[
     PDF[NormalDistribution[0, 1], x] 
     Log[PDF[NormalDistribution[0, 1], x]], 
     {x, a, b}]/
    Integrate[PDF[NormalDistribution[0, 1], x], {x, a, b} ]])

which took around 2.5 seconds to run on a MacBook Air.