Deriving Bayesian Predictive Distribution?

83 Views Asked by At

Given $ p(\mathbf{w} | \mathbf{t}, \alpha, \beta) = \mathcal{N}\left(\mathbf{w} | \mathbf{m}_{N}, \mathbf{S}_{N}\right)$ and $p(t | \mathbf{w}, \beta) = \mathcal{N}\left(t | \mathbf{w}^{\mathrm{T}} \boldsymbol{\phi}\left(\mathbf{x}\right), \beta^{-1}\right)$ and $ p(t | \mathbf{t}, \alpha, \beta)=\int p(t | \mathbf{w}, \beta) p(\mathbf{w} | \mathbf{t}, \alpha, \beta) \mathrm{d} \mathbf{w} $

How can I obtain the following form: $p(t | \mathbf{x}, \mathbf{t}, \alpha, \beta)=\mathcal{N}\left(t | \mathbf{m}_{N}^{\mathrm{T}} \boldsymbol{\phi}(\mathbf{x}), \sigma_{N}^{2}(\mathbf{x})\right) $ with $ \begin{aligned} \mathbf{m}_{N} &=\mathbf{S}_{N}\left(\mathbf{S}_{0}^{-1} \mathbf{m}_{0}+\beta \mathbf{\Phi}^{\mathrm{T}} \mathbf{t}\right) \\ \mathbf{S}_{N}^{-1} &=\mathbf{S}_{0}^{-1}+\beta \boldsymbol{\Phi}^{\mathrm{T}} \boldsymbol{\Phi} \\ \sigma_{N}^{2}(\mathbf{x}) &=\frac{1}{\beta}+\boldsymbol{\phi}(\mathbf{x})^{\mathrm{T}} \mathbf{S}_{N} \boldsymbol{\phi}(\mathbf{x}) \end{aligned} $


Here is my attempt: $P(t | t, \alpha, \beta)=\int P(t | w, \beta) \cdot P\left(w | t_{1} \alpha, \beta\right) dw \\ = \int N\left(t | w^{\top} \phi, \beta^{-1}\right) N\left(w | m_{N}, S_{N}\right) dw \\ = \frac{1}{(2 \pi)|\beta|^{-1 / 2}\left|S_{N}\right|^{1 / 2}} \int {e^{ -\frac{1}{2}\left(t-w^{\top} \phi\right)^{\top} \beta\left(t-w^{\top} \phi\right)-\frac{1}{2}\left(w-m_{N}\right)^{\top} S_{N}^{-1}\left(w-m_{N}\right)}} dw \\ \approx \frac{1}{(2 \pi)|\beta|^{-1 / 2}\left|S_{N}\right|^{1 / 2}} \sum_{w} e^{ -\frac{1}{2}\left(t-w^{\top} \phi\right)^{\top} \beta\left(t-w^{\top} \phi\right)-\frac{1}{2}\left(w-m_{N}\right)^{\top} S_{N}^{-1}\left(w-m_{N}\right)} $

Focusing on the exponential term, we can expand, re-arrange terms, complete the square and obtain:

$e^{ -\frac{1}{2}\left(t-w^{\top} \phi\right)^{\top} \beta\left(t-w^{\top} \phi\right)-\frac{1}{2}\left(w-m_{N}\right)^{\top} S_{N}^{-1}\left(w-m_{N}\right)} \\ = e^{-\frac{1}{2}\left(w^{\top} \left( \phi^{\top} \beta \phi + S_{N}^{-1} \right)w\right)} e^{-\frac{1}{2}\left(2 w^{\top} \phi^T \beta t - 2 w^{\top} S_{N}^{-1} m_{N}\right)} e^{-\frac{1}{2}\left(t^{T} \beta t+m_{N}^{T} S_{N}^{-1} m_N\right)} \\ = e^{-\frac{1}{2}\left(w^{\top} \left( \phi^{\top} \beta \phi + S_{N}^{-1} \right)w\right)} e^{-\frac{1}{2}\left(2 w^{\top} \left( \beta \phi^T t - {S_N}^{-1} m_{N} \right) \right)} e^{-\frac{1}{2}\left(t^{T} \beta t+m_{N}^{T} S_{N}^{-1} m_N\right)} $


$\therefore e^{ -\frac{1}{2}\left(t-w^{\top} \phi\right)^{\top} \beta\left(t-w^{\top} \phi\right)-\frac{1}{2}\left(w-m_{N}\right)^{\top} S_{N}^{-1}\left(w-m_{N}\right)} \\ \propto e^{-\frac{1}{2}\left(w^{\top} \left( \phi^{\top} \beta \phi + S_{N}^{-1} \right)w\right)} e^{-\frac{1}{2}\left(\beta \phi^T t - {S_N}^{-1} m_{N}\right)} $

Let ${\Sigma_M}^{-1} = \phi^T\phi\beta + {S_N}^{-1}$ and let $\mu_M = \Sigma_M \left( \beta \phi^T t - {S_N}^{-1} m_{N} \right)$

$\therefore P(t | t, \alpha, \beta) \approx \sum_{w} N\left(t, w | \mu_M, \Sigma_M \right)$