Probability distribution over probability distributions

Question

This may not be a well defined question, but I couldn't find anything about it. Perhaps, I am looking with wrong keywords. Anyway, suppose $X$ is a set, possibly a subset of an Euclidean space. Define $\Delta(X)$ as the set of probability measures over $X$. Similarly define $\Delta(\Delta(X))$. Now the claim that I am struggling with is the following: Take $\mathcal{P}\in \Delta(\Delta(X))$ then we can find $P\in \Delta(X)$. I can see that we can define $P(A)=\int_{\Delta(X)} f(A) \mathcal{P}(df)$ for any ``appropriate'' $A$, but I want to really understand this claim. For example, I don't know what kind of integral $\int_{\Delta(X)}$. Is this thing, $P(A)$ is always well defined? If you suggest me any book, lecture notes or paper that would be great. Many thanks in advance!

Answer 1

To make things a bit more rigorous, you are going to need some $\sigma$-fields. Let $\cal A$ be a $\sigma$-field on $X$, and define $\Delta(X)$ to be the space of all probability measures on $(X,{\cal A})$.

Now let ${\cal F}$ be the $\sigma$-field on $\Delta(X)$ generated by the collection of maps $f\mapsto f(A)$ for $A\in{\cal A}$.

For any probability measure $\cal P$ on $(\Delta(X),{\cal F})$ you can define $P\in\Delta(X)$ by
$$P(A)=\int_{\Delta(X)}f(A) \,{\cal P}(df)\quad\mbox{ for }A\in{\cal A}.\tag1$$ This is the usual integral of a random variable on the probability space $(\Delta(X),{\cal F},{\cal P})$. You can check that (1) defines a genuine probability measure on $(X,{\cal A}).$

Intuitively, $P$ is the mean value of the random probability measure with law $\cal P$.