Non conjugate prior with known posterior

Question

For a give likelihood function $p(x | \theta)$, the prior $p(\theta)$ is a conjugate prior if the posterior $p(\theta | x)$ comes from the same family of distributions as $p(\theta)$.

$p(\theta | x) = \frac{p(x | \theta)p(\theta)}{p(x)}$ involves solving an (often intractable) integral so this is very useful.

Are there any prior, likelihood pairs such that the posterior's distribution is known but not from the same family as the prior? Meaning, non conjugate priors where the posterior is still easy to find. If so, how is this called?

Answer 1

This is going to depend on how broadly you are prepared to define conjugate family.

For example if you have a Bernoulli likelihood with parameter $p$, the conjugate family is made up of the Beta distributions.

• As a different prior, we could imagine that the prior in an example is rather more restricted, say

  • $\mathbb P_0(p=\frac14)=\frac35$ and
  • $\mathbb P_0(p=\frac34)=\frac25$.

• we now sample $5$ times and see the result of $3$ successes and $2$ failures, with likelihood proportional to $p^3(1-p)^2$

• and this gives priors times likelihoods proportional to

  • $\frac35 \times \left(\frac14\right)^3 \times \left(\frac34\right)^2$ and
  • $\frac25 \times \left(\frac34\right)^3 \times \left(\frac14\right)^2$

• so the posterior distribution becomes

  • $\mathbb P_{3 \text{ of } 5}(p=\frac14)=\frac13$ and
  • $\mathbb P_{3 \text{ of } 5}(p=\frac34)=\frac23$

which is simple enough. Something similar would happen with any prior distribution over a finite number of possible values for $p$ and any observation, producing a posterior distribution over the same possible values for $p$.

In this example the posterior distribution is in a similar form to the prior distribution. Does that mean they are both part of a non-Beta conjugate family of distributions?