Equivalent ways of writing Kullback-Leibler divergence

170 Views Asked by At

Let $P$ and $Q$ denote distributions functions of a continuous random variable. Then the Kullback-Leibler divergence between $P$ and $Q$ is defined as: $$ D_{KL}(P\Vert Q):=\int_{-\infty}^{+\infty} \log\left(\frac{p(x)}{q(x)}\right)p(x)dx $$ where $p(x)=\partial P(x)/\partial x$ and $q(x)=\partial Q(x)/\partial x$. Now, according to Wikipedia's page on Kullback-Leibler divergence

https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

it seems that also $$ D_{KL}(P\Vert Q)=\int_{-\infty}^{+\infty} \log\left(\frac{P(x)}{Q(x)}\right)P(x)dx, $$ see the second point in the section "Properties". Yet, I do not see why the two expressions shall coincide. As an example, let $P(x)=\exp(-\sigma/x)$, $Q(x)=\exp(-\lambda/x)$, $x>0$, with $\sigma, \lambda>0$ - i.e. Fréchet dustribution. Then, on one hand $$ \int_{-\infty}^{+\infty} \log\left(\frac{p(x)}{q(x)}\right)p(x)dx=\log(\sigma / \lambda)-(\sigma-\lambda)\int_0^{+\infty}t\sigma e^{-t\sigma}dt\\ =\log(\sigma / \lambda) -1+\frac{\lambda}{\sigma}. $$ On the other one $$ \int_{-\infty}^{+\infty} \log\left(\frac{P(x)}{Q(x)}\right)P(x)dx=-\frac{\sigma-\lambda}{\sigma}\int_0^{+\infty}\frac{\sigma}{x}e^{-\sigma/x}dx, $$ where the integral of the right hand side equals the expectation of a random variable distributed according to $P$ and, as such, diverges. So in this case the two expressions do not coincide. Am I missing something?

Probably in the Wikipedia page there is some abuse of notation and the author intended to use densities in both expressions. If so, I am wondering: is there a way to express $D_{KL}(P\Vert Q)$ in terms of the cdf's $P$ and $Q$ rather than in terms of the densities $p$ and $q$?