Show that $KL_X(\theta;\psi) = (\theta-\psi)^2/2$.

Question

Suppose that $Q$ and $P$ are two probability measures with densities $p$ and $q$ w.r.t. a common measure $\nu$.

The Kullback-Leibler information of $Q$ from $P$ is defined by $$KL(\theta\,;\psi) = \int\log\dfrac{p(x)}{q(x)}p(x)d\nu(x).$$ In case of parametric families, where $X\sim P_\theta$ and $P_\theta \ll \nu$ for all $\theta$ with densities $f_X(\cdot|\theta)$, we write $$KL(\theta\,;\psi) = \operatorname{E}_\theta\log\dfrac{f_X(X|\theta)}{f_X(X|\psi)}, \,\,\theta,\psi \in \Omega.$$

Exercise: If $X\sim N(\theta,1)$, show that $KL_X(\theta\,,\psi) = (\theta - \psi)^2/2.$

What I've tried to do: If I'm not mistaken the normal distribution is part of the exponential family, and I need to use the second part of the definition. I get $KL_X(\theta\,,\psi) = \operatorname{E}_\theta\dfrac{(x-\theta)^2}{(x-\psi)^2}$, which would mean that $KL_X(\theta\,,\psi) = 0$.

Questions:

• I'm not sure whether I fully grasp the concept of probability measures and densities. This definition states that the probability measures have a density of their own. But don't we speak of densities of distributions with respect to their probability measures? Furthermore, what would a common measure $\nu$ be? What would the probability measures in this exercise be?

• How do I solve this exercise?

Answer 1

We have that $KL(\theta\,;\psi) = \operatorname{E}_\theta\log\dfrac{f_{X|\theta}}{f_{X|\psi}}$, and we know that for $X|\theta \sim N(\theta,1)$ the density is given by: \begin{equation} f_{X|\theta} = \frac{\textrm{Exp}[-\frac{1}{2}(x-\theta)^2]}{\sqrt{2\pi}} \end{equation}

Then the log of the ratio of the two densities is then:

\begin{equation} \begin{split} \textrm{log}\frac{f_{X|\theta}}{f_{X|\psi}} & = \textrm{log}f_{X|\theta} - \textrm{log}f_{X|\psi} \\ & = \frac{1}{2}[-(x-\theta)^2 + (x-\psi)^2]\\ & = -\frac{1}{2}[-x^2 + 2x\theta -\theta^2 + x^2 - 2x\psi +\psi^2] \end{split} \end{equation}

Taking the expectation with the distribution under $\theta$, we have $E[x] = \theta$ and $E[x^2] = 1+\theta^2$.

Then:

\begin{equation} \begin{split} -\frac{1}{2}[- x^2 + 2x\theta -\theta^2 + x^2 - 2x\psi +\psi^2] & = -\frac{1}{2}[-1-\theta^2 + 2\theta^2 -\theta^2 + 1 + \theta^2 - 2\theta\psi + \psi^2]\\ & = \frac{1}{2}[\theta^2 -2\theta\psi +\psi^2]\\ & = \frac{(\theta-\psi)^2}{2} \end{split} \end{equation}