Does there exist a Wasserstein-like measure for difference between probability distributions, where the space on which the probabilities are defined is not connected?
E.g. a distribution over six states; three of these have defined distances with respect to one another (such as [0 1 2;1 0 1;2 1 0]); and the other three as well; but the two sets of three do not have distances defined with respect to one another.
Is there some Wasserstein-something-else hybrid that has been explored (or something else) that addresses this situation?
thank you!
I think the generalization you're probably looking for is the more general transportation cost/distance, of which the Wasserstein distance is just a special case.
Take any 2 probability spaces $(X,\mu)$ and $(Y,\nu)$. Define a transportation cost
$$ c(x,y):X\times Y\rightarrow [0,\infty] $$ This represents the cost of transporting 1 "unit" of mass from $X$ to $Y$. Then, you can immediately write down the generalized transportation distance
$$ T_c(\mu,\nu) = \inf_{\pi\in\Pi(\mu,\nu)}\int c(x,y)d\pi $$ where $\Pi(\mu,\nu)$ is the set of joint probability measures on $X\times Y$ such that the marginals are $\mu$ and $\nu$ i.e. $\int_X d\pi = d\nu$, $\int_Y d\pi = d\mu$. This is the generalization of the Wasserstein distance.
The definition of the cost function is entirely up to you (though some will be easier to work with than others!) The Wasserstein distance chooses $c(x,y) = d(x,y)^p$ in the case when both probability measures on defined on a common metric space, because this is convenient and allows us to prove more theorems. You can prove pretty general things without this assumption, though - see the two books by Villani and another two by Rachev + Ruschendorf, or maybe User's guide to optimal transport, a set of notes on Arxiv by Ambrosio. For instance, as soon as $X$ and $Y$ are individually complete separable metric spaces and the cost is lower semicontinuous, you can define the transportation distance and it has nice(ish) properties. See chapter 4 in Villani's Optimal Transport, Old and New (available on his website, I believe).
From a more philosophical point of view, you might ask yourself the following question: if the two spaces "aren't connected", how are you going to measure the cost of transporting material from one to the other? You really need a "path" to get from one to the other in order to transport mass. Otherwise the transportation cost should always be infinite (intuitively). If you can answer "why" the two sets are disconnected, this might help you define a logical cost function. Otherwise I don't see how a transportation distance even makes sense.