solution to this probability paradox?

Question

This paradox must probably already have been answered on math.stackexchange, but I can't find it.

Say we have a standard normally distributed variable $X$ (or any other distribution really).

Then we have an interval $I$ (e.g.$(-1,1)$), for which the probability that $X$ is in $I$ is positive. So when we sample $X$, we could for example get $0.345....$

Yet for any number $r \in \mathbb R$, the probability that $X$ will turn out to be equal to $r$ is $\int_r^rD_X(x)dx=0$ Where $D_X$ is the probability density function of $X$.

Paradox: So for any individual $r\in R$, the probability $P(X=r)=0$. That means that when we sampled $0.345...$ just now, an event occurred that has probability $0$ of occurring.

How is this paradox resolved?

Answer 1

$0$ probability doesn't imply an impossible event. If you think of probability of an event as the ratio of number of occurrences of the event to the total number of occurrences, then as the continuous random variable can take uncountably many values, the probability that it is equal to a particular number is negligible or $0$. This is not mathematically rigorous but that is the intuition.

Answer 2

This paradox is resolved when you realize that finite and infinite probability spaces don't always act the same. In any probability space, for some event $E$, $pr(E)=0$ means that the probability measure of $E$ is zero. In finite spaces we have that this implies that $E$ never happens. This same implication doesn't hold for infinite probability spaces since the union of infinitely many sets of measure zero can have positive measure.

Answer 3

Abishanka's and Sean's intuitions are good and correct. Here are related thoughts:

1) You may as well make your random variable $X$ uniformly distributed over the interval $[0,1]$. Each point is equally likely, there are infinitely many, so the probability of each must be zero. We must integrate the uniform density over an interval in order to accumulate any positive probability.

Same with throwing darts at a dart board: Each point is equally likely but there are infinitely many of them. We can only talk about positive probabilities of the dart falling in a region with positive area.

So if you first pick a particular number $x \in [0,1]$, then $P[X=x]=0$. If you pick a countably infinite set $\mathcal{A} \subsetneq [0,1]$, then $P[X \in \mathcal{A}]=0$. Notice that we are picking the point and/or the set first, and then wondering if we will win the "probability 0 lottery" that $X$ lands on our point or set. On the other hand, we know that some outcome will occur, so $P[X \in [0,1]]=1$.

2) Since $X$ is uniformly distributed over $[0,1]$, its value will be irrational with probability 1. This is described by an infinite decimal precision. Do you really think you could specify such a number in advance? An actual computer simulation would truncate $X$ to finite precision, in which case the approximation $\tilde{X}$ is discrete and all of its (finite number) of outcomes have positive probability.

3) We can only sum over countably infinite sets: The interval $[0,1]$ contains an uncountably infinite number of points. So if we try to obtain contradictions by a summing argument, we will fail, since our summation cannot sum over all possible outcomes.

A related interesting theorem is the following:

Theorem:

Let $B$ be an uncountably infinite set. Let $f:B\rightarrow\mathbb{R}$ be a function with positive values (so that $f(x)>0$ for all $x \in B$). Then there is a countably infinite subset $C \subsetneq B$ such that $\sum_{x \in C} f(x) = \infty$.

Proof:

Let $\mathcal{F}$ be the collection of all finite subsets of $B$. Define $m = \sup_{A \in \mathcal{F}} \sum_{x \in A} f(x)$. Then there is a sequence of finite sets $\{A_n\}_{n=1}^{\infty}$ such that $A_n \in \mathcal{F}$ for all $n$, and $\lim_{n\rightarrow\infty}\sum_{x \in A_n} f(x) = m$. The set $\cup_{i=1}^{\infty} A_i$ is the countable union of a finite sets, so it is either finite or countably infinite. Clearly: $$ \sum_{x \in A_n} f(x) \leq \sum_{ x \in \cup_{i=1}^{\infty} A_i} f(x) \quad, \forall n \in \{1, 2, 3, ...\} $$ and taking limits as $n\rightarrow \infty$ gives $m \leq \sum_{ x \in \cup_{i=1}^{\infty}A_i} f(x)$. If $m=\infty$, then we are done, since we just define the finite or countably infinite set $C = \cup_{i=1}^{\infty} A_i$ and we have $C \subsetneq B$ and $\sum_{x \in C} f(x) = \infty$. (In particular, since the sum is infinite, the "finite or countably infinite" set $C$ cannot be finite, and so it is countably infinite.)

Now suppose $m < \infty$ (we reach a contradiction). Since $\cup_{i=1}^{\infty} A_i$ is a finite or countably infinite subset of the uncountably infinite set $B$, there is a point $y \in B$ such that $y \notin \cup_{i=1}^{\infty} A_i$. Note that $f(y)>0$. Now choose an index $k$ such that $\sum_{x \in A_k} f(x) \geq m-f(y)/2$. Note that $y \notin A_k$. Consider the finite set $A_k \cup \{y\}$. Then $\sum_{x \in A_k \cup \{y\}} f(x) = f(y) + \sum_{x \in A_k} f(x) \geq f(y) + m-f(y)/2 > m$, contradicting the fact that the sum over any finite subset of $B$ must be less than or equal to $m$ (by definition of $m$). $\Box$

In particular, this theorem implies that any probability distribution can have an at-most-countably-infinite number of positive point masses. It also shows why we are only allowed to sum over a countably infinite number of terms: Summing an uncountably infinite number of positive numbers necessarily gives $\infty$.