KL-Divergence of Uniform distributions

5.6k Views Asked by At

Having $P=Unif[0,\theta_1]$ and $Q=Unif[0,\theta_2]$ where $0<\theta_1<\theta_2$

I would like to calculate the KL divergence $KL(P,Q)=?$

I know the uniform pdf: $\frac{1}{b-a}$ and that the distribution is continous, therefore I use the general KL divergence formula: $$KL(P,Q)=\int f_{\theta}(x)*ln(\frac{f_{\theta}(x)}{f_{\theta^*}(x)})$$ $$=\int\frac{1}{\theta_1}*ln(\frac{\frac{1}{\theta_1}}{\frac{1}{\theta_2}})$$ $$=\int\frac{1}{\theta_1}*ln(\frac{\theta_2}{\theta_1})$$

From here on I am not sure how to use the integral to get to the solution.

1

There are 1 best solutions below

2
On BEST ANSWER

You got it almost right, but you forgot the indicator functions. So the pdf for each uniform is $$P(P=x) = \frac{1}{\theta_1}\mathbb I_{[0,\theta_1]}(x)$$ $$\mathbb P(Q=x) = \frac{1}{\theta_2}\mathbb I_{[0,\theta_2]}(x)$$

Hence, $$ KL(P,Q) = \int_{\mathbb R}\frac{1}{\theta_1}\mathbb I_{[0,\theta_1]}(x) \ln\left(\frac{\theta_2 \mathbb I_{[0,\theta_1]}}{\theta_1 \mathbb I_{[0,\theta_2]}}\right)dx $$

Since $\theta_1 < \theta_2$, we can change the integration limits from $\mathbb R$ to $[0,\theta_1]$ and eliminate the indicator functions from the equation. Also, since the distribution is constant, the integral can be trivially solved $$ \int_{\mathbb R}\frac{1}{\theta_1}\mathbb I_{[0,\theta_1]} \ln\left(\frac{\theta_2 \mathbb I_{[0,\theta_1]}}{\theta_1 \mathbb I_{[0,\theta_2]}}\right)dx = \int_{\mathbb [0,\theta_1]}\frac{1}{\theta_1} \ln\left(\frac{\theta_2}{\theta_1}\right)dx=$$ $$ =\frac {\theta_1}{\theta_1}\ln\left(\frac{\theta_2}{\theta_1}\right) - \frac {0}{\theta_1}\ln\left(\frac{\theta_2}{\theta_1}\right)= \ln\left(\frac{\theta_2}{\theta_1}\right) $$ And you are done.