Gibbs sampling for t distribution with location and scale paameter

Question

I have observations $X_{1},..,X_{100}$. Further $X_{i} \sim t(\mu, \sigma, \nu)$, where $\mu$ is location parameter, $\sigma$ is scale parameter and $\nu$ is degree of freedom. I want Gibbs samples from the posterior distribution $P(\mu, \sigma, \nu|X)$. How do I develop the sampler?

Thank you very much in advance!

Answer 1

Assuming that what you want to do is sample from the posterior distribution for the distribution parameters $P(\mu, \sigma, \nu | \mathbf{X})$, where $\mathbf{X}$ is your set of observations $\mathbf{X} = X_1, X_2, ..., X_{100}$, then you would go about it in the following way:

Constructing the posterior

First we need to construct the joint-likelihood of all your observations as follows:

$$ P(\mathbf{X} | \mu, \sigma, \nu) = \prod_{i} P(X_i | \mu, \sigma, \nu), $$ where $P(X_i | \mu, \sigma, \nu)$ is the student-t density.

Now we can define the posterior through Bayes' theorem: $$ P(\mu, \sigma, \nu | \mathbf{X}) = \frac{P(\mathbf{X} | \mu, \sigma, \nu) P( \mu, \sigma, \nu) }{P(\mathbf{X})} $$

Gibbs sampling algorithm

First, choose some initial guess for the parameters $\theta_0 = (\mu_0, \sigma_0, \nu_0)$.

The update $\theta_n \to \theta_{n+1}$ is performed as follows:

1. Sample $\mu_{n+1}$ from $P(\mu | \sigma_n, \nu_n , \mathbf{X})$
2. Sample $\sigma_{n+1}$ from $P(\sigma | \mu_{n+1}, \nu_n , \mathbf{X})$
3. Sample $\nu_{n+1}$ from $P(\nu | \mu_{n+1}, \sigma_{n+1}, \mathbf{X})$
4. Let $\theta_{n+1} = (\mu_{n+1}, \sigma_{n+1}, \nu_{n+1})$

This is the Gibbs sampling update.

Parameter conditional distributions

Clearly this requires you to sample from the 1D parameter conditional distributions. It may be that the conditionals can be derived analytically for the sudent-t distribution for some specific choices of prior, but this is not necessary - you can still evaluate them numerically.

As an example of how you derive conditionals, here is the procedure for $P(\mu | \sigma, \nu , \mathbf{X})$. The product rule allows us to write $$ P(\mu, \sigma, \nu | \mathbf{X}) = P(\mu | \sigma, \nu, \mathbf{X}) P( \sigma, \nu | \mathbf{X}), $$ where $P( \sigma, \nu | \mathbf{X})$ is a marginal distribution given by $$ P( \sigma, \nu | \mathbf{X}) = \int_{-\infty}^{\infty} P(\mu, \sigma, \nu | \mathbf{X}) \, \mathrm{d}\mu. $$ Solving for $P(\mu | \sigma, \nu, \mathbf{X})$ we can write $$ P(\mu | \sigma, \nu, \mathbf{X}) = \frac{P(\mu, \sigma, \nu | \mathbf{X})}{P( \sigma, \nu | \mathbf{X})} = \frac{P(\mathbf{X} | \mu, \sigma, \nu) P( \mu, \sigma, \nu)}{\int_{-\infty}^{\infty} P(\mathbf{X} | \mu, \sigma, \nu) P( \mu, \sigma, \nu) \, \mathrm{d}\mu} $$

To actually draw samples from the conditional distributions, typically you would use either rejection sampling or 1D metropolis-hastings MCMC.

1D Metropolis-Hastings MCMC

Suppose you want to sample values of a variable $x$ some 1D conditional distribution $P(x | I)$:

Metropolis-Hastings update: $x_n \to x_{n+1}$
1. Sample a proposed update $\hat{x}$ from a normal proposal distribution $\mathcal{N}(x_n, \sigma_x)$
2. Sample a random value $t \sim \mathrm{Uniform}(0,1)$
3. if $t < P(\hat{x} | I) / P( x_n | I)$, we accept the proposal and set $x_{n+1} = \hat{x}$, else repeat steps 1 & 2 until a proposal is accepted.

Here $\sigma_x$ is the proposal width, and for this approach to be efficient we want to choose $\sigma_x$ such that about 50% of proposed steps are accepted. Most implementations of MCMC have an internal mechanism to adjust $\sigma_x$ towards an optimal vlaue.

Given that you are new to this area, you may still have some questions - if you edit your question to ask them I'll do my best to answer.