Dont see why this linear algebra equation is correct

43 Views Asked by At

From the wikipedia page on Bayesian Linear Regression

A prior $\rho(\beta, \sigma^2)$ is conjugate to this likelihood function if it has the same functional form with respect to $\beta$ and $\sigma$. Since the log-likelihood is quadratic in $\beta$, the log-likelihood is re-written such that the likelihood becomes normal in $(\beta - \hat{\beta})$. Write

$$ (\mathbf{y} - \mathbf{X}\beta)^T (\mathbf{y} - \mathbf{X}\beta) = (\mathbf{y} - \mathbf{X} \hat{\beta})^T (\mathbf{y} - \mathbf{X}\hat{\beta}) + (\beta - \hat{\beta}) \mathbf{X}^T\mathbf{X} (\beta - \hat{\beta}) $$

It kind of just throws this out without showing why it is true.

1

There are 1 best solutions below

0
On

First do a Taylor expansion (the function is quadratic in $\beta$): $$ (y-X\beta)^T(y-X\beta) = (y-X\hat\beta)^T(y-X\hat\beta) + 2 (y-X\hat\beta)^TX(\beta-\hat \beta) + (\beta-\hat\beta)^TX^TX(\beta-\hat\beta).$$ Since $\hat\beta$ satisfies $X^TX\beta = X^Ty$, the middle term vanishes. This is because $\hat\beta$ is a least-squares solution, i.e., it solves $$ \min_\beta (y-X\beta)^T(y-X\beta) $$