Finding a distribution with Bayesian statistics

Question

Suppose $X$ is a random variable that follows a binomial distribution with parameters $n = 5$ and $p$ where $0 \leq p \leq 1$ is unknown. Through a Bayesian approach, $p$ is modeled with a random variable $P$. Some researchers have established that it is very likely that $p$ is close to $1$ and not likely that $p$ is close to $0$.

(i) Which of the following densities is the best choice of prior for $P$: $f(p) = 1, 0 \leq p \leq 1$, $g(p) = 2p, 0 \leq p \leq 1$, or $h(p) = e^{-p}, p \geq 0$. Justify.

(ii) Let $f_P(p)$ be the prior you chose in (i) and you observe that $X = 5$. Given this observation, find the posterior density $P$.

(iii) Between the prior and posterior densities of $P$ in (ii), which has more density near $1$? Explain why the density you identify should have more density near $1$?

My attempt:

(i) I am not quite sure which prior is the best. I eliminated the third one $h(p)$ due to the support being too large. Then, the first one $f(p)$seems to assume that it is always $1$, which doesn't seem consistent. The third one also seems most likely as it only multiplies the values by $2$ which corresponds to the fact that the researchers already established that $p$ is close to $1$.

I also recall that usually, the prior and posterior of a binomial tend to be beta. The general beta distribution support is $0$ to $1$ which eliminates $h(p)$.

(ii) The chosen prior is $f_P(p) = 2p, 0 \leq p \leq 1$

The pdf of $X$ is:

$$f(x\mid p) = \frac{5!}{x!(5-x)!} p^x (1-p)^{5-x},$$

$x = 0, 1, 2, 3, 4, 5$.

The posterior is then:

$$f(p\mid x) = \frac{f(x\mid p)\cdot f_P(p)}{f_X(x)} = \frac{f(x\mid p)\cdot f_P(p)}{f_X(x)} = \frac{f(x\mid p)\cdot f_P(p)}{\int_{-\infty}^\infty f(x\mid p) \cdot f_P(p) \,dp}$$

We apply proportionality:

$$\frac{f(x\mid p)\cdot f_P(p)}{\int_{-\infty}^\infty f(x\mid p) \cdot f_P(p) \,dp} \propto f(x\mid p)\cdot f_P(p) = f(x\mid p) = \frac{5!}{x!(5-x)!} p^x (1-p)^{5-x} \cdot 2p = \frac{2\cdot 5!}{x!(5-x)!} p^{x + 1} (1-p)^{5-x}$$

As I stated before, the posterior of a binomial is often beta, and both the appearance of the pdf and support seem to agree with this.

So I conclude that the posterior distribution is Beta with parameters $\alpha$ as $x + 2$ and $\beta$ as $6 - x$.

The posterior pdf: $f(p\mid x) = \frac{\Gamma (8)}{\Gamma (x + 2) \Gamma (6 - x)} p^{x + 1} (1 - p)^{5 - x}$, $0 < p < 1$.

Letting $X = 5$, $f(p\mid 5) = \frac{\Gamma (8)}{\Gamma (7) \Gamma (1)} p^{6} (1 - p)^{0} = \frac{\Gamma (8)}{\Gamma (7) \Gamma (1)} p^{6}$

(iii) Not sure about this one.

Any assistance is much appreciated. Also, if anyone can provide a better explanation for (i), that would be great.

Answer 1

Your bottom line for the posterior density is correct.

I don't know what you mean by saying "it only multiplies the values by $2$ which corresponds to the fact that the researchers already established that $p$ is close to $1.$" But that density is large when $p$ is near $1$ and small when $p$ is near $0.$ I also don't understand what you mean by "doesn't seem consistent." If you meant it's not consistent with the requirement of a large probability that $P$ is near $1$ and a small probability that $P$ is near $0,$ then it is correct, but unclearly expressed.

I would write the conditional probability mass function of $X$ given that $P=p$ as $f_{X\,\mid\,P\,=\,p}(x),$ but many write it as $f_X(x\mid P=p).$ The notation $f(x\mid p)$ I find obnoxious, but it is unfortunately widely used. Note that $f_X(3)$ and $f_P(0.8)$ are clear, but what is $f(x\mid p)$ in the case where $x=3$? Is it $f(3\mid p),$ i.e. the conditional density of some random variable when $x=3$? Which random variable? The notation $f_{X\,\mid\,P\,=\,p}(3)$ is clear about that.

Rather than $= \frac{5!}{x!(5-x)!} p^x (1-p)^{5-x} \cdot 2p,$ I would write $\propto p^x (1-p)^{5-x} \cdot 2p,$ and state explicitly that this means proportional as a function of $p,$ not as a function of $x$ or of anything else, and that the expression involving $x$ is constant as a function of $p$ because the variable $p$ does not appear within it. Then I would deal with the normalizing constant only at the end, based on what is known about the beta distribution. That is simpler than carrying along the expressions involving $x$ and evaluating the integral, since the values of the integrals needed to find the normalizing constant for the beta density have already been done before you come to this problem.

In part (iii), note the the datum, $X=5,$ is more probable if $p$ is near $1$ than it isn't, so you would expect the posterior to put even more probability near $1$ than the prior does.

Answer 2

Remarks:

(i) You are correct. The prior $f(p)$ is useless as it does not account for the non-uniform anticipation. The prior $h(p)$ in fact contradicts the expectation that $p$ lies closer to $1$, as $h(p)$ is a strictly decreasing function.

(ii) You can significantly simplify your working as follows. Since the p.d.f. of $X$ is $$f(x\mid p)=\binom 5xp^x(1-p)^{5-x},$$ the posterior distribution is given by $$f(p\mid x)\propto f(x\mid p)\cdot f_P(p)=C(x)p^{x+1}(1-p)^{5-x}$$ where $C(x)$ is independent of $p$. When $x=5$ we have $f(p\mid x=5)=Cp^6$ and as the posterior distribution is a density, we conclude that $$f_P(p)=2p;\quad f(p\mid x=5)=7p^6.$$

(iii) Evidently the posterior has much greater weighting near $1$ as $p^6\ll p$ when $p$ is small. This is in line with Bayesian theory, as the posterior should usually be an improvement of the prior distribution derived from an elicitee's expectations. Further, as Michael has mentioned, the observed data point is at the maximum value, and we can check from the p.d.f. of $X$ that the value of $p$ is more likely to be near $1$.