Conjugate priors: wht not binomial-binomial?

Question

Citing from Kevin Murphy's machine learning book:

When the prior and the posterior have the same form, we say that the prior is a conjugate prior for the corresponding likelihood. Conjugate priors are widely used because they simplify computation, and are easy to interpret, as we see below. In the case of the Bernoulli, the conjugate prior is the beta distribution [...]

The bold "the" is mine, and it is my source of doubt: would another binomial distribution be a valid conjugate prior for a binomial distribution, since the result would be a binomial distribution as well (I would add "of course", but I'm not sure of anything at this point!)?

EDIT: Oops, maybe I got it: the $\theta$ parameter in the binomial distribution would not be the variable of the prior if I used a binomial distribution as prior as well. In other words, $\text{Bin}(N_1, N_2\mid\theta)$ is the binomial distribution describing the likelihood. I cannot use the same distribution as a prior because I need a distribution where $\theta$ is the "variable" (I don't know which better term to use, in contrast with "parameter"), such as $\text{Beta}(\theta\mid a,b)$. Is this correct?

Answer 1

Another binomial distribution would not be a valid conjugate prior except in the utterly trivial case where it's a Bernoulli distribution, taking only the values $0$ and $1$. This is trivial because it means we have a coin that either always turns up heads or always tails. We toss the coin once and then we know that all subsequent outcomes will be the same. A binomial distribution including numbers bigger than $1$ in its support cannot serves as a prior for a parameter that must be in the interval $[0,1]$.

However, there are other conjugate priors for the family of binomial distributions. One must realize that it is a family of distributions, not just one distribution, that is conjugate. Start with one distribution whose support is a subset of $[0,1]$, and close under the operation of multiplying by likelihood functions and then normalizing, and you've got a family of conjugate priors. Unless of course, having the "same form" is construed so as to exclude that, but "same form" is actually a vague phrase.

Notice that every beta distribution is a continuous distribution, and the binomial distribution is discrete. What does "same form" mean then?

Answer 2

You have a bit of confusion about what a conjugate prior means. You have a family of distributions with some parameter distinguishing different members of the family; in your example, $p$ is that parameter in the binomial distribution $$ f(x) = \binom{n}{x}p^x(1-p)^{n-x}$$

You have some prior distribution for the value of that parameter. Let's say the prior for $p$ were $\frac{1}{4}$ times a binomial distribution with $n^*=2$ and $p^* = \frac{1}{3}$. Then the posteriori distribution would be a funky sum of three different binomiial-looking pieces, and not itself a binomial distribution.

Answer 3

A prior distribution is conjugate if the posterior distribution belongs to the same family, but the sampling distribution need not belong to the same family (though in some cases, it could).

If the sampling distribution is binomial or Bernoulli, then the parameter $\theta$ representing the individual probability of success can be regarded as a random variable with prior distribution on the support $\theta \in [0,1]$. The distribution of $\theta$ obviously cannot be binomial nor Bernoulli itself for this reason.

However, a case for which the conjugate prior belongs to the same family as the sampling distribution is the Normal-Normal case; i.e., the samples $x_1, \ldots, x_n$ belong to a normal distribution and if the mean is itself normal then the posterior mean is also normal.

Why are conjugate priors desirable? The idea is that if the choice of prior distribution is such that the posterior belongs to the same parametric family, then this allows for the convenient calculation of the posterior distribution for new data. For example, in the Bernoulli/Beta case, once you have observed a sample of some number of observations, you can calculate the posterior distribution of the parameter $\theta$, which captures your belief about the true value of $\theta$ given the information received so far. But if you continue the experiment and collect more Bernoulli observations, updating your posterior distribution does not require recalculation from the original sample--you only need the previously known posterior and set that as your prior. This property is what makes Bayesian parameter estimation particularly attractive.