What does a probability of $1$ mean?

Question

From a textbook on probability on the Law of Large Numbers:

Theorem 3-19 (Law of Large Numbers): Let $X_1,X_2, \ldots , X_n$ be mutually independent random variables (discrete or continuous), each having finite mean and variance. Then if $S_n = X_1 + X_2 +\dots+ X_n$,

$$ \lim_{n \to\infty} P\left(\left|\frac{S_n}{n} - \mu\right| \geq \varepsilon\right) = 0 $$

Since $S_n$ is the arithmetic mean of $X_1,X_2, \ldots , X_n$, this theorem states that the probability of the arithmetic mean $\frac{S_n}{n}$ differing from its expected value $\mu$ by more than $\varepsilon$ approaches zero as $n \to \infty$. A stronger result, which we might expect to be true, is that $ \lim_{n \to\infty} \frac{S_n}{n} = \mu $ but this is actually false. However, we can prove that $ \lim_{n \to\infty} \frac{S_n}{n} = \mu $ with probability one.

The only difference between the last sentence and the one before that is the phrase 'with probability one'. What does probability one mean here ? The usual definition is that the event occurs with 100% certainty. If that is the case, why is the original assertion $ \lim_{n \to\infty} \frac{S_n}{n} = \mu $ false ?

Answer 1

Probability zero does not mean "never happens", and probability one does not mean "always happens". For example, if you choose a real number uniformly in the interval $[0,100]$, you will choose an integer with probability zero, and a transcendental number with probability one. That doesn't mean there aren't any integers though.

To understand what "probability zero" means, you need to know some measure theory; it means that the measure of that event is zero. Similarly, "probability one" means that the measure of that event is one. This idea is used quite frequently in probability theory, because measure-zero events are often annoying and we want to not think about them.

Answer 2

Doing it below with random variables — same idea than in your case (where it s the event `"something happens to that sequence of random variables")

The difference between $X=a$ and $X=a$ a.s. (almost surely, i.e. with probability one) is better understood when you actually see a random variable as what is it: a function from some set $\Omega$ equipped with a probability distribution to some set of values $\mathcal{X}$.

"$X=a$" means "$\forall\omega\in\Omega,\ X(\omega)=a$", i.e "$\{\omega\in \Omega: X(\omega)=a\}=\Omega$".

However, "$X=a$ a.s" (or, equivalently, $\mathbb P\{X=a\}$) only means the measure of the set $\{\omega\in \Omega: X(\omega)=a\}$ is 1. There might be some $\omega$'s for which $X(\omega)\neq a$; but the measure of the set of such $\omega$'s is $0$ (that is, the set of "bad" $\omega$'s is negligible).

Answer 3

Throw a dart at a square target, the probability of its landing in any subregion of the square being proportional to the area of the region.

The probability that it fails to land exactly on the diagonal from the upper left corner to the lower right corner is $1$, since the area of that diagonal is zero.

But that doesn't mean every point in the space of points where it could land fails to be on that diagonal. Some points are on the diagonal.

That's the difference.

Answer 4

$S_n$ is itself a random variable. Lets say $\mu$ is 0.5. $S_n = 0$ is a possible event iff $X_i=0$ is possible for every i. That means that $lim_{n\rightarrow ∞}\frac{S_n}{n}$ is also a random variable! It is $\mu$ with probability one, but there are events for which it is not $\mu$!