Bayesian normal posterior in linear regression

67 Views Asked by At

I'm trying to show the following:

Assume that $Y_i \sim \mathcal{N}(\beta x_i, 1)$, $i=1, \dots, n$ are independent random variables. Concretely, we consider a regression model $Y_i = \beta x_i + \varepsilon_i$, $\varepsilon_i \sim \mathcal{N}(0, 1)$ with fixed variance and covariate $x_i$.

Let $\hat{\beta}$ be the maximum likelihood estimator of $\beta$. It is known that

$$ \hat{\beta} = \frac{\sum_{i=1}^n x_iy_i}{\sum_{i=1}^n x_i^2} $$

and that $\hat{\beta}|\beta \sim \mathcal{N}\left(\beta, \frac{1}{\sum_{i=1}^n x_i^2} \right)$.

Let $\beta \sim \mathcal{N}(0,1)$.

Show that the distribution of $\beta | \hat{\beta}$ is

$$ \mathcal{N}\left(\frac{\sum_{i=1}^n x_iy_i}{1 + \sum_{i=1}^n x_i^2 }, \frac{\sum_{i=1}^n x_i^2}{1 + \sum_{i=1}^n x_i^2 } \right) $$

Heres my work so far:

We know that

\begin{align*} \pi(\beta | \hat{\beta}) &\propto \pi(\beta) f(\hat{\beta}|\beta) \propto \exp\left(-\frac{\beta^2}{2} - \frac{\sum_{i=1}^n x_i^2}{2} (\hat{\beta}-\beta)^2 \right)\\ &\propto \exp\left(-\frac{1}{2} \left( \left(1 + \sum_{i=1}^n x_i^2 \right)\beta^2 - 2 \sum_{i=1}^n x_i^2 \beta \hat{\beta} \right) \right)\\ & = \exp\left(-\frac{1+\sum_{i=1}^n x_i^2}{2} \left( \beta^2 - \frac{2 \sum_{i=1}^n x_i^2 \beta \hat{\beta}}{1+\sum_{i=1}^n x_i^2} \right) \right)\\ &= \exp\left(-\frac{1+\sum_{i=1}^n x_i^2}{2} \left( \beta^2 - \frac{2 \sum_{i=1}^n y_ix_i \beta }{1+\sum_{i=1}^n x_i^2} \right) \right) \end{align*}

but already here I'm getting the wrong variance. I'm missing a $\sum_{i=1}^n x_i^2$. Can anyone spot the error?

1

There are 1 best solutions below

0
On BEST ANSWER

I think the error is in the problem statement. If you take all the $x_i$ to $\infty$ while the variance of the $\epsilon_i$ stays constant, you can estimate $\beta$ with arbitrary precision, so the variance should go to zero, as yours does and theirs doesn't.