Separating Distributions with A Plane

Question

Is the following statement true? If so does there exist a proof or something similar?

"Given two independent probability distributions, $P_{A}(X)$ and $P_{B}(X)$, neither of which is uniform, there is defined at least one distance and direction in that space at which the two distributions could be centred and be simultaneously separated by a flat plane within an accepted tolerance."

Suppose I have two Gaussian distributions placed side by side. To my mind, there exists a distance at which the means are seperated enough that could draw a line between them and expect the bulk of the integral to sit either side of the line. A silly example might be the flat plane that exists in my frame of reference between my computer and I.

I imagine two uniform distributions are inseparable, since both distributions are constant over all the space they are defined in.

I am afraid I am not a mathematician, thus I have no conception of whether this is a rational statement. I would be interested to know if there are examples where this is not true. Is there a more elegant way of rendering what is meant by this statement?

Answer 1

I do not really know how to formalize your statement. If you allow yourself to do any translation in $\mathbb{R}^d$, then you can trivially separate uniform distributions over compact sets by just moving them so that their support don't intersect.

---

A related problem that arises in statistics is the problem of clustering. A typical setting (the Gaussian mixture model) is the following: given $k$ centers $\mu_1,\ldots,\mu_k\in\mathbb{R}^d$, you are given $X_1,\ldots,X_n$ independently sampled from the following procedure:

• choose $i$ uniformly at random in $\{1,\ldots,k\}$
• return $X\sim \mathcal{N}_d(\mu_i,I_d)$

The goal is to estimate $\mu_1,\ldots,\mu_n$ (we also consider the related task of finding, for each $j$, from which of the $k$ Gaussian distributions $X_j$ has been sampled). You can view this as trying to separate empirically each center. The interesting question is not qualitative anymore (is it possible to separate the centers?), but quantitative (how many samples do you need for that?).