What Happens to the Posterior Probability as Both Variance and Amount of Data Approach Infinity

Question

$\DeclareMathOperator{\Pr}{Pr}\DeclareMathOperator{\Nd}{N}$ I was reading the first chapter Andrew Gelman's book Bayesian Data Analysis (3rd edition), and I was doing the first exercise in the book, of which the setup is this:

Suppose that if $\theta = 1$, $y$ has a normal distribution $\Nd(\mu=1, \sigma)$, and if $\theta = 2$, then $y$ has a normal distribution $\Nd(\mu=2, \sigma)$. Also suppose $\Pr(\theta = 1) = 0.5$ and $\Pr(\theta = 2) = 0.5$.

Part C of the question was this: what happens to $\Pr(\theta = 1 | y = 1)$ as $\sigma \to \infty$?
The answer is that the probability that theta is one or two approaches $0.5$, which means we get essentially no information from any data.

My question is this: what happens if we had a theoretically infinite amount of data? So suppose $Y$ was infitite, what happens to $\Pr(\theta = 1 | Y)$ as $\sigma \to \infty$?

Answer 1

Well, it depends. Let $Y = (y_1, y_2, \dotsc)$ be an infinite sequence (of "observations", or "data"), and let $Y_N = (y_1, y_2, \dotsc, y_N)$. Then you are essentially asking what happens to the posterior distribution $p(\theta | Y_N)$ as $N$ and $\sigma^2$ approach infinity simultaneously (whatever that means). The answer will depend on two things:

1. What the data $y_1, y_2, \dotsc$ are. As usual (in Bayesian data analysis) we are conditioning on the data, which are assumed given, and the expectation of the posterior will depend on the given data.
2. How exactly you let $N$ and $\sigma^2$ approach infinity. If you take $\sigma^2 = \ln N$ and let $N \to \infty$, for example, then the variance of the posterior will converge to $0$, while if you take $\sigma^2 = e^N$ it will diverge to $\infty$.

Specifically, one can use Bayes' formula to work out that

$$p(\theta | Y_N) \propto \exp\left[ - \frac{N}{2 \sigma^2} \left( \theta - \bar y_N \right)^2 \right],$$

where $\bar y_N$ is the sample mean

$$\bar y_N = \frac{1}{N} \sum_{n=1}^N y_n.$$

That is, $\theta | Y_N \sim \mathcal N(\bar y_N, \sigma^2/N)$. (Note that I'm using the more common notation $\mathcal N(\textit{mean}, \textit{variance})$, rather than $\mathcal N(\textit{mean}, \textit{standard deviation})$.)

If we make the very reasonable assumption that the sequence $\bar y_1, \bar y_2, \dotsc$ of sample means converges to a value $\bar y$ then this is indeed what the expectation of $\theta | Y_N$ converges to as $N \to \infty$. This is not very interesting.

The variance on the other hand can be made to converge to any non-negative value, or to not converge at all. We previously mentioned ways to make it approach 0 and $\infty$, and you are now in a position to verify those results. As another example, consider taking $\sigma^2 = \lambda N$ for some $\lambda > 0$, and note that $\theta | Y_N \sim \mathcal N(\bar y_N, \lambda)$ independently of $N$, and certainly in the limit $N \to \infty$.