What does it mean for something to be closed under sampling?

Question

I have this statement and I'm having trouble understanding what it means for the parameter to be closed under sampling.

Answer 1

This is a definition in Bayesian Inference. In general, we have for a given sample $x$:

$$p(\theta|x) = \frac{p(\theta)p(x|\theta)}{\int_{\theta \in \Theta}p(\theta)p(x|\theta)}$$

Normally, there is no guarantee that $p(\theta)$ and $p(\theta|x)$ belong to the same family. But if $p(\theta)$ is the conjugate prior to the likelihood function $f(x|\theta)$ then they do, which is a nice algebraic/analytical convenience.

Examples (extensive list here):

If we are looking at coin flips, our probability model is often a binomial distribution (assuming iid coin flips). Bayesian inference allows us to choose any prior for $\pi = P(H)$ that takes $[0,1]$ as its domain. However, if we choose the Beta distribution to represent our prior information about $\pi$, then the posterior $p(\theta|x)$ will also be a Beta distribution.

Therefore, if we assume Beta prior with a Binomial sampling model, then we will get Beta posterior.

Abstracting out one level, if we define $S_{\mathcal{G}}(\mathcal{F})$ as the posterior distribution after drawing a sample from some distribution $g \in \mathcal{G}$ using a prior $f\in \mathcal{F}$ then we see that if $\mathcal{F}$ is conjugate to $\mathcal{G}$ then:

$$S_{\mathcal{G}}(\mathcal{F}) \in \mathcal{F}$$

Which makes $\mathcal{F}$ closes under the sampling operation $S_{\mathcal{G}}(\mathcal{F})$