How do we make the "intuitive calculation" of conditional probability, rigorous?

Question

The definition of Conditional Probability for events $A$ and $B$ in sample space $S$ is $$\mathbb{P}(A|B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}.$$

Sometimes, we use a rearranged version of this formula to calculate the probability of the intersection of events - called the multiplicative law of probability:

$$\mathbb{P}(A\cap B)=\mathbb{P}(A|B)\times \mathbb{P}(B)$$

When using this formula, how does one calculate $\mathbb{P}(A|B)?$ Since by definition the intersection is required to find the conditional probability? Is there an alternative definition/way to compute the conditional probability when you don't know the intersection?

I have calculated the conditional probability through intuition many times (E.g. picking Red/Blue marbles out of a bag, without replacement), but I was wondering if there was some sort of standard convention on how calculate the conditional probability when you don't know the intersection?

Example.

Say we have three people (Alex, Bob, Carol) with their three hats. Say I take all their hats, mix them up, and then return one to each person. What is the probability that person A and B get exactly their own hat back?

"Solution": The way I would think of it is: Let $E_A$ and $E_B$ be the events that Alex and Bob get their hats back respectively. Then, $$\mathbb{P}(E_A\cap E_B)= \mathbb{P}(E_B)\times \mathbb{P}(E_A|E_B)$$

The probability of $E_B$ would be $\frac{1}{3}$. Now, the way I would calculate $\mathbb{P}(E_A|E_B) $ intuitively, even though I don't know what the intersection is (because that's what I'm trying to find), is "Since Bob has his hat, I have two hats left, which gives a probability of $\frac{1}{2}$ for Alex to get his hat back."

This intuitive logic of getting to the conditional probability directly, when I didn't use/bypassed the definition, is what I would like to clarify/formalise.

Answer 1

This doesn't answer the question exactly, but is a bit much for a comment:

Sometimes the conditional probability is already known. For example, let $\{X_n:n=0,1,2,\ldots\}$ be a Markov chain on the nonnegative integers with initial distribution $\alpha$ and transition matrix $P$, that is, for each nonnegative integer $i$ we have $\mathbb P(X_0=i)=\alpha_i$ and for each pair of nonnegative integers $i,j$ we have $$ \mathbb P(X_{n+1} = j\mid X_n = i) = P_{ij}, $$ (the $(i,j)$-entry of $P$). Then, the distribution of $X_1$ would be given by $$ \mathbb P(X_1 = j) = \sum_{i=0}^\infty \mathbb P(X_1 = j\mid X_0=i)\mathbb P(X_0=i) = \sum_{i=0}^\infty P_{ij}\alpha_i. $$

Answer 2

Say we have a bag with a red marble and two blue marbles. We draw two marbles. What is the probability they're both different colors? If we draw red on the first turn, we have a 100% chance of drawing different colors. If we draw blue, we have a 50% chance. So we have a $\frac23$ chance of drawing different overall. Clearly, I've used the conditional distribution of the event "drawing different colors" given the color of the first marble. How did I do it?

It turns out that formally, the answer is fairly boring. I did it by assuming that the conditional distributions are as I say they are in the very statement of the problem. When you talk about "drawing without replacement", you're basically saying the following:

1. There is a set $M=\{r, b_1, b_2\}$ of marbles.
2. There are two marble-valued random variables, $X_1$ and $X_2$.
3. $X_1$ is uniform on $M$.
4. $X_2$ is uniform on $M\setminus\{X_1\}$.

This isn't a proper mathematical description. The problem is point 4. When you describe a random variable as being "uniform on $A$", $A$ has to be a set. But $M\setminus\{X_1\}$ is not a set, it's a set-valued random variable. So how do I give a formal meaning to point 4? All I can really do is just directly assert that $X_2$ has the conditional distribution I expect it to have:

4. The conditional distribution of $X_2$ given $X_1=x$ is uniform on $M\setminus\{x\}$.

Answer 3

There is not one formula, but there is a process.   You are using this process, and it is correct to do so when you can.

You should have some model for measuring your probability, and you might be able to adjust this model to evaluate a conditional probability.   If you can do so, then you may do so.

As long as your model is consistent, it will not matter which approach you take -- calculate the joint from the conditional and a marginal, or a conditional from the joint and a marginal -- though one route may seem easier than another … and that's why we often use Bayes' Rule after all.

---

For instance, in your bag of red and blue marbles example, your model is that each marble in the bag (at the time of drawing) is equally likely to be selected (that there is no bias), so you may therefore evaluate probabilities using the counts of marbles of each colour in the bag.

$$\textsf{The conditional probability that the second marble is red, when given that the first is blue,}\\\textsf{assuming that there were originally $m$ red and $n$ blue marbles in the bag, will be}\\\mathsf P(R_2\mid B_1)=\dfrac{m}{m+n-1}\\\textsf{The probability that we first draw a red marble and secondly a blue,}\\\textsf{ when drawing two marbles from the bag without replacement, is:}\\\mathsf P(B_1\cap R_2){=\dfrac{\binom n1\binom m1/2!}{\binom{m+n}{2}}\\=\dfrac{n\cdot m}{(m+n)(m+n-1)}}$$

---

Your hat example is similar, and your thinking is correct.   If Bob happens to have his hat when the hats were distributed without bias, then Alan either had his own hat or Carol's with equal probability.

Alternatively: When the hats are distributed without bias, there are six equally probable ways to do so, among which only one way will give Alan and Bob their own hats. However, there are two ways among the six that Bob could have his own hat back.$$\mathsf P(E_A\cap E_B)=\tfrac{1}{3!}\\\mathsf P(E_B)=\tfrac {2!}{3!}\\\therefore\quad\mathsf P(E_A\mid E_B)=\tfrac 1{2!}$$