Marginalization question $\Pr[a] = E_X[\Pr[\ a|X\ ]\ ]$

Question

I'm reading explanation of a theorem, and there's one step that I can't understand. I know it should be simple enough, but I just can't think of the reasoning atm.

The step says,

According to Marginalization, $\Pr[a] = E_X[\Pr[\ a|X\ ]\ ]$

Why is this true? What is the meaning of Marginalization? Also, I know expected values of a random variable, but what does it mean to have an expected value of a probability $(\Pr[a|X])$?

Answer 1

The way I would consider it is, instead of the random event $a$, to consider the indicator random variable $I_a$ which takes the value $1$ when $a$ occurs and $0$ when it does not. For example $$E[I_a]=\Pr(a).$$

The law of total expectation gives $$E[I_a] = E_X[\,E[I_a|X ]\, ]$$ which corresponds to the result you state.

Answer 2

$\mathsf P(a\mid X)$ is the name of a function mapping values of the random variable to a marginal probability measure.

$$\mathsf P(a\mid X)(x) = \mathsf P(a\mid X=x)$$

It's similar to the way we sometimes write a probability mass function as: $\mathsf P_X(x) = \mathsf P(X=x)$

For a discrete random variable the expression becomes:

$$\begin{align} \mathsf P(a) & = \mathsf E_X[\mathsf P(a\mid X)] \\ &= \sum_{x\in\chi} \mathsf P(a\mid X=x)\mathsf P(X=x) \\ & = \sum_{x\in\chi} \mathsf P(a, X=x) \end{align}$$

This is just the Law of Total Probability in another guise.

And for a continuous random variable this is:

$$\begin{align} \mathsf P(a) & = \mathsf E_X[\mathsf P(a\mid X)] \\ &= \int_{\chi} \mathsf P(a\mid X=x)\,f_X(x)\operatorname d x \end{align}$$