I have two discrete random variables $X$ and $Y$ and their distributions have different support. Assume $X$ and $Y$ can both take on the same number of values. Lets say $X$ takes values in $\{10,13,15,17,19\}$ and $Y$ takes values in $\{12,14,16,18,20\}$.
I would like to use the Kullback-Leibler Divergence but it requires that Q dominates P. Is it possible to modify the support of each random variable so that they have the same support?
If not, are there any measures of statistical distance that do not require $X$ and $Y$ to have the same support?
One solution I have created is to make kernel density estimators with a gaussian kernel using the datasets collected on $X$ and $Y$. Now the densities $\hat{f}(x)$ and $\hat{g}(y)$ have support on $( -\infty, \infty)$ and with suitable bandwidth they are multimodal with modes centered around the support of the original random variables. It remains to be seen how wise or foolish of an idea this is.
Note: Since the KL divergence of a finite gaussian mixture does not have a closed form solution, I used monte carlo methods to estimate it.
Statistically, Kullback-Leibler divergence measures the difficulty of detecting that a certain distribution say H1 is true when it is thought initially that a certain other distribution say H0 is true. It essentially gives the so-called Bahadur slope for this problem of discrimination (i.e. of testing). If, as in your example, each distribution has some support on a set for which the other distribution has no support then perfect inference becomes possible and the divergence is legitimately infinite. The more interesting case is where one support is fully contained in the other. In that case one of the hypotheses can be confirmed with certainty while the other only exponentially fast so the divergence will be infinite only in one direction.