Hard posteriori distribution

34 Views Asked by At

Let $y_1, \ldots, y_n$ be independent observations of $Y_i \sim \mathcal{N}(\beta x_i, \sigma^2), i = 1, \ldots, n$, where $x_i$ and $\sigma^2$ are known, and $\beta$ is an unknown parameter. Consider the prior $\beta \sim \mathcal{N}(\beta_0, m^2)$ and find the posterior distribution of $\beta$.

$ \pi(\beta | \mathbf{y}) \propto (2\pi)^{-\frac{n}{2}} (\sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (y_i - \beta x_i)^2\right] (2\pi)^{-\frac{1}{2}} (m^2)^{-\frac{1}{2}} \exp\left[-\frac{1}{2m^2} (\beta - \beta_0)^2\right] \propto \exp\left[-\frac{1}{2\sigma^2} \sum_{i=1}^{n} (y_i - \beta x_i)^2\right] \exp\left[-\frac{1}{2m^2} (\beta - \beta_0)^2\right] $

I was doing this exercise and tried many times to go further than what's above, but I can't get anywhere, the expected answer is below. Is there a better way to proceed that doesn't involve opening the squares and performing "algebraic miracles"?

Answer:

$\frac{\beta}{y} \sim \mathcal{N}(\beta_1, \sigma_1^2); \quad \sigma_1^2 = \frac{1}{{\left(\frac{1}{\sigma_2}\sum_{i} x_i^2\right) + \frac{1}{m}}}$; $\beta_1 = \sigma_1^2 \left( \frac{\sum_{i} (y_i x_i)}{\sigma^2} + \frac{\beta_0}{m^2} \right)$

1

There are 1 best solutions below

0
On BEST ANSWER

The posterior is proportional to $$\begin{align} f(\beta \mid \boldsymbol x, \boldsymbol y) &\propto e^{-(\beta - \beta_0)^2/(2m^2)} \prod_{i=1}^n \exp \left(-\frac{(y_i - \beta x_i)^2}{2\sigma^2}\right) \\ &= \exp \left(-\frac{(\beta - \beta_0)^2}{2m^2} - \frac{1}{2\sigma^2} \sum_{i=1}^n (x_i \beta - y_i)^2 \right). \tag{1} \end{align}$$ We must now collect like terms in powers of $\beta$: $$\begin{align} -\log f &\propto \left(\frac{1}{2m^2} + \frac{1}{2\sigma^2} \sum_{i=1}^n x_i^2 \right) \beta^2 - \left( \frac{\beta_0}{m^2} + \frac{1}{\sigma^2} \sum_{i=1}^n x_i y_i \right) \beta + C \\ \tag{2} \end{align}$$ where $C$ is some constant that gives rise to a multiplicative constant of proportionality to $f$ and can be ignored. If we let $A$ be the coefficient of $\beta^2$ and $B$ be the coefficient of $\beta$ in $(2)$, then completing the square would yield $$A \beta^2 + B \beta = A \left(\beta^2 + \frac{B}{A} \beta \right) = A \left( \beta + \frac{B}{2A} \right)^2 - \frac{B}{2A}.$$ So again, we can ignore the trailing constant $-\frac{B}{2A}$ and we recognize that $f$ is proportional to a normal density of the form $$f(\beta \mid \boldsymbol x, \boldsymbol y) \propto \exp \left(-\frac{(\beta + \frac{B}{2A})^2}{\frac{1}{A}}\right); \tag{3}$$ that is to say, the posterior of $\beta$ is normal with posterior mean $$-\frac{B}{2A} = \left(\frac{\beta_0}{m^2} + \frac{1}{\sigma^2} \sum_{i=1}^n x_i y_i\right)\left(\frac{1}{m^2} + \frac{1}{\sigma^2} \sum_{i=1}^n x_i\right)$$ and posterior variance $$\frac{1}{2A} = \frac{1}{m^2} + \frac{1}{\sigma^2} \sum_{i=1}^n x_i,$$ as claimed. Note that you have a typographical error in your posterior variance: the correct term is $\frac{1}{m^2}$, not $\frac{1}{m}$. The takeaway from this exercise is that you should not waste effort keeping track of constants of proportionality with respect to $\beta$. The factors containing $2\pi$ and so forth, are all independent of $\beta$ and can be ignored. Work with the kernel of the posterior and the computations are much simpler.