KL-divergence with zero probability?

1.1k Views Asked by At

Suppose $P(X=1)=P(X=2)=1/2$ and $P(Y=1)=1$. Then $$D(Y||X)=\log\left(\frac{1}{1/2}\right)+0\cdot\log\left(\frac{0}{1/2}\right).$$ If we consider $$\lim_{p\to0}p\log(2p)=\lim_{p'\to\infty}\frac{\log(2/p')}{p'}=\lim_{p'\to\infty}-\frac{1}{p'}=0$$

So in this case it seems like things with a probability of zero can just be dropped. Is that always the case? It seems like it should be, but I wanted to check.

1

There are 1 best solutions below

0
On BEST ANSWER

Nevermind, I see that wikipedia states this as being true.