Suppose that I have the following Bayesian network:
I want to calculate $p(\omega | \mathbf{x})$, where $\mathbf{x} = [x_{1}, \ldots, x_{N}]$ is my observed data. My distributions are:
$$ \mathbf{x}_{n} \sim \mathcal{N}\left( \mu, \sigma^{2} \right) \; i.i.d.$$ $$ \sigma^{2} \sim \mathcal{IG}\left( s_{1}, s_{2} \right) $$ $$ \mu \sim \mathcal{N}\left( \omega, \tau^{2} \right) $$ $$ \tau^{2} \sim \mathcal{IG}\left( t_{1}, t_{2} \right) $$ $$ \omega \sim \mathcal{N}\left( \omega_{0}, V_{0} \right) $$
where $\mathcal{N}(\cdot)$ and $\mathcal{IG}(\cdot)$ are the normal and inverse gamma distributions respectively. So in other words:
$$ p(\mathbf{x}, \sigma^{2}, \mu, \tau^{2}, \omega) = p(\mathbf{x} | \mu, \sigma^{2}) p(\sigma^{2}) p(\mu | \omega, \tau) p(\omega) p(\tau^{2}) $$
I can do my desired calculation via Bayes rule:
$$ p(\omega | \mathbf{x}) = \frac{p(\omega, \mathbf{x})}{p(\mathbf{x})} = \frac{p(\mathbf{x} |\omega) p(\omega)}{p(\mathbf{x})} $$
My question is -- what are the expressions for $p(\mathbf{x})$ and $p(\mathbf{x} |\omega) $
Expression for $p(\mathbf{x})$:
If I assume iid, then I know:
$$ p(\mathbf{x}) = \prod_{n=1}^{N} p(x_{n}) $$
But what is $p(x_{n})$? Well, we just need to marginalize over all the other variables, right? Ie
$$p(x_{n}) = \int_{U} \int_{\Sigma} p(x_{n} | \mu, \sigma^{2}) p(\mu | \omega, \tau) p(\sigma^{2}) d \mu d \sigma^{2} $$
But how do I take into account the fact that $\omega, \tau$ are also random variables? I need to include them in the integration somehow? So does it become:
$$p(x_{n}) = \int_{U} \int_{\Sigma} \int_{\Omega} \int_{T} p(x_{n} | \mu, \sigma^{2}) p(\mu | \omega, \tau^{2}) p(\sigma^{2}) p(\omega) p(\tau^{2}) d\omega d\tau^{2} d \mu d \sigma^{2} $$
Expression for $p(\mathbf{x}| \omega)$:
This is the same thing, except now I need to condition on $\omega$. So does that mean:
$$p(x_{n} | \omega) = \int_{U} \int_{\Sigma} \int_{T} p(x_{n} | \mu, \sigma^{2}) p(\mu | \omega, \tau^{2}) p(\sigma^{2}) p(\tau^{2}) d\tau^{2} d \mu d \sigma^{2} $$
Or what?
Also, how can I compute these distributions? These integrals seem quite intractable if I have them correctly. Could I somehow approximate them or sample from them directly?
Questions:
- What is the analytical expression for $p(\mathbf{x})$? I.e. have I setup my integration correctly?
- What is the analytical expression for $p(\mathbf{x} | \omega)$? I.e. have I setup my integration correctly?
- How can I actually compute these distributions in practice?
Sorry for the basic questions. I am trying to learn how to work with these type of networks on my own and have nobody to ask.