Relative entropy between singular measures

Question

Usually, to define relative entropy between two probability measures, one assumes absolute continuity. Is it possible to extend the usual definition in the non absolutely continuous case?

Answer 1

Relative entropy between two probability measures $P$ and $Q$ can be defined even if $P$ is not absolutely continuous w.r.to $Q$. In any case, $P$ and $Q$ are absolutely continuous w.r.to a common measure $\mu$ (one can take $\mu$ to be $\frac{P+Q}{2}$). Then relative entropy between $P$ and $Q$ is defined as $$D(P\|Q)=\int p\log\frac{p}{q}d\mu,$$ where $p=dP/d\mu$ and $q=dQ/d\mu$.