Bayseian Inference: Asymptotic Normal Approximation

Question

The Question:

Given random variables $X_1,\dots,X_n \overset{i.i.d.}{\sim}\text{Ber}(\theta)$, where $\theta \in (0,1)$ has a $\text{Beta}(a,b)$ prior distribution $$\pi(\theta)=\frac{1}{B(a,b)}\theta^{a-1}(1-\theta)^{b-1}.$$

(i) Obtain the normal approximation distribution to the posterior distribution.

(ii) Verify that the mean and variance of the approximation are approximately $\dfrac rn$ and $\dfrac{r(n-r)}{n^3}$ respectively, where $r=\sum\limits_{i=1}^n x_i$.

---

My Attempt:

So I started by computing the MLE of $\theta$, which we know is $$\hat \theta = \frac rn$$

The density function for Bernoulli is $$f(x_i)=\theta^{x_i}(1-\theta)^{1-x_i}$$

The likelihood function is then $$L(\theta)=\theta^r(1-\theta)^{n-r}$$

The log likelihood is $$l(\theta)=\ln[L(\theta)]=r\ln(\theta)+(n-r)\ln(1-\theta)$$

The observed information is \begin{align} \ & J(\theta)=-\frac{d^2l}{d\theta^2}=\frac{r}{\theta^2}+\frac{n-r}{(1-\theta)^2} \\ \ \implies & J(\hat \theta) = \frac{r}{(r/n)^2}+\frac{n-r}{(1-r/n)^2}=\frac{n^3}{r(n-r)} \end{align}

Hence the normal approximation is $$\theta\mid\vec x \approx \text N(\hat \theta,J(\hat \theta)^{-1})=\text N \biggl(\frac rn, \frac{r(n-r)}{n^3} \biggr)$$

I have not used the beta prior at all, and from what question (ii) is asking, I have clearly misunderstood something...

Answer 1

1. Since given one observation $X = x$, the posterior distribution of $θ$ is ${\mit Β}(a + x, b + 1 - x)$, then given $n$ obersavtions $x_1, \cdots, x_n$, the posterior distribution of $θ$ is$$ {\mit Β}\left(a + \sum\limits_{k = 1}^n x_k, b + n - \sum\limits_{k = 1}^n x_k\right) = {\mit Β}(a + r, b + n - r). $$ Because the mean and the variance of ${\mit Β}(a + r, b + n - r)$ are $\dfrac{a + r}{a + b + n}$ and $\dfrac{(a + r)(b + n - r)}{(a + b + n)^2(a + b + n + 1)}$, respectively, then the normal approximation of posterior distribution is$$ N\left( \frac{a + r}{a + b + n}, \frac{(a + r)(b + n - r)}{(a + b + n)^2(a + b + n + 1)} \right). $$
2. For large $n$, because $r ≈ n θ_0$, where $θ_0$ is the true value of $θ$, then$$ \frac{a + r}{a + b + n} = \frac{\dfrac{a}{n} + \dfrac{r}{n}}{\dfrac{a + b}{n} + 1} ≈ \dfrac{r}{n}, $$\begin{align*} \frac{(a + r)(b + n - r)}{(a + b + n)^2(a + b + n + 1)} &= \frac{\left(\dfrac{a}{n} + \dfrac{r}{n}\right)\left(\dfrac{b}{n} + 1 - \dfrac{r}{n}\right)}{n\left(\dfrac{a + b}{n} + 1\right)^2\left(\dfrac{a + b}{n} + 1 + \dfrac{1}{n}\right)}\\ &≈ \frac{\dfrac{r}{n} \left( 1 - \dfrac{r}{n} \right)}{n} = \frac{r(n - r)}{n^3}. \end{align*}