Why is the formula $f_{X|Y}(x|y) = \frac{P_{Y|X}(y|x) f_X(x)}{P_Y(y)}$ true?

71 Views Asked by At

Suppose we have a random sample $X=(X_1,\dots, X_n)$ which depends on $\theta$, treated as a value of the random variable $\Theta$. The posterior pdf is given by $$f_{\Theta | X}(\theta|x)=\frac{f_{X|\Theta}(x|\theta)f_\Theta(\theta)}{\int_{-\infty}^\infty f(x,\theta)d\theta}$$

where $f_\Theta$ is the prior pdf of the random variable $\Theta$.

All $f$s in this case are pdfs, and it is clear where this formula is coming from: we just need to express $f(x,\theta)$ in two different ways and set them equal to each other.

But what if $f_\Theta$ is a pdf while $f_{X|\Theta}$ is a pmf? For example, maybe the random sample in this case is the result of some number of coin flips. In that case, each $X_i$ is a discrete random variable, and $f_{X|\Theta}$ should be a pmf rather than a pdf.

At the same time, $f_\Theta$ could be something like $f_\Theta(\theta)=\text{const}\cdot \theta^2(1-\theta)^2$ for some $\text{const}$ which makes it a pdf.

So my question is: if in the displayed formula above, $f_\Theta$ and $f_{\Theta|X}$ are pdfs but $f_{X|\Theta}$ is a pmf, then the formula takes the form

$$f_{\Theta | X}(\theta|x)=\frac{p_{X|\Theta}(x|\theta)f_\Theta(\theta)}{\int_{-\infty}^\infty f(x,\theta)d\theta}.$$

And if so, how to prove such a formula that has both pdfs and pmfs in it? It also becomes unclear what $f(x,\theta)$ should mean in this context.

Addition: in this note, the author is essentially saying that (in my notation) $f_{X|\Theta}(x|\theta)$ is

a conditional probability or a conditional density depending on whether $X$ is a continuous or discrete random variable

But again, what is the justification for the formula when $X$ is discrete?..

Addition 2: This page also says (without giving a reason) that in my first displayed formula at the very top, we can replace pdf with pmf for one of the variables.