Posterior distribution intractability and accept-reject, MCMC sampling

Question

I am a little confused about whether accept-reject sampling can be used to sampling from intractable posterior distributions, $f(x) = P(\theta | x)$.

Given a posterior, $P(\theta | x) = \frac{P(x | \theta)P(\theta)}{\int P(x | \theta)P(\theta)}$, in many cases the denominator is intractable. Because we cannot compute the integral, we cannot compute the probabilities associated with some $\theta$ and $x$.

In MCMC, this problem is solved by using an acceptance criterion that cancels the denominator. In accept-reject sampling however, the acceptance criterion is $u < \frac{f(x)}{Cg(x)}$ where $g(x)$ is our proposal function and $C$ the scaling constant that ensures $f(x) < Cg(x)$ (and $u \sim \operatorname{Uniform}(0, 1)$).

This seems like we are required to calculate $f(x)$, and so requires a tractable posterior distribution. However, I have seen that accept-reject sampling can be used for intractable distributions.

Answer 1

Rejection sampling works even if $f(x)$ and $g(x)$ are only known up to a constant of proportionality. This can be seen as absorbing these constants of proportionality into $C$. A question from stats.stackexchange.com addresses this, and that site may well be a better venue for similar questions.

You can thus sample from the posterior by using the unnormalised posterior density, $P(x|\theta)P(\theta)$, and the proposal distribution PDF alone, which are all assumed to be easy to calculate in standard Bayesian statistics.

Answer 2

Another way to look at this is to say that you’re not scaling $g$ by $C$, but $f$; you’re defining a new density $f/C$ that’s upper-bounded by $g$ and isn’t normalized, but scaled such as to fit under $g$ as tightly as possible. Then it’s clear that a multiplicative constant in $f$ is irrelevant, since you’re rescaling it anyway.