Visualizing the Weak Law of Large Numbers

Question

I've understand in layman terms what is stated by the weak law of large numbers:

$P(\left | \frac{X_{1}+...+X_{n}}{n} \right |-\mu \geq \epsilon )\rightarrow 0 $ as $n\rightarrow +\infty $

Unfortunatly, i'm not able to visually imagine the shape of the random variable $\left | \frac{X_{1}+...+X_{n}}{n} \right |-\mu $. Can you help me showing how this random variable takes shape?

Answer 1

You probably intended $P\left(\left | \frac{X_{1}+...+X_{n}}{n} -\mu\right | \geq \epsilon \right) \to 0$ : otherwise consider the case with negative $\mu$

If each of the $X_i$ has finite variance $\sigma^2$ then, in a Central Limit Theorem sense, $\left | \frac{X_{1}+...+X_{n}}{n} -\mu\right |$ could be approximated for large $n$ by a half-normal distribution with scale parameter $\frac{\sigma}{\sqrt{n}}$, i.e. with a density function which looks like the positive half of a bell curve which narrows as $n$ increases

Answer 2

Some simulations per request in comment:

Convergence of running sample averages to population mean is shown for three distributions. Speed of convergence of $A_n$ to $\mu$ depends on the standard deviation of the population (kept roughly constant here). All graphs show high variability at the left (for small sample sizes), but 'settle down' to better behavior at the right (for large sample sizes). Examples, programmed in R statistical software, are easily adaptable to other distributions

Normal data: Let $X_1, X_2, ..., X_{5000}$ be independent and distributed as $\mathsf{Norm}(100, 15); \sigma = 15.$ Look at a graph of $A_n = \bar X_n = \frac 1 n \sum_{i=1}^n X_i$ against $n,$ for $n = 1, \dots, m = 5000.$ [In this case, $\bar X \sim \mathsf{Norm}(\mu, \sigma/\sqrt{n})$ exactly. But that is a separate issue from convergence of $\bar X$ to $\mu$ according to LLN.]

m = 5000; x = rnorm(m, 100, 15); n = 1:m; a = cumsum(x)/n
plot(n, a, type="l", lwd=2, col="blue", ylim=c(90,110))
abline(h = 100, col="green3")

Exponential data: Let $X_1, X_2, ..., X_{5000}$ be independent and distributed as $\mathsf{Exp}(rate=1/15), \sigma = 15.$ Look at a graph of $A_n = \bar X_n = \frac 1 n \sum_{i=1}^n X_i$ against $n.$ [For very large $n,\, A_n - \mu$ is approximately normal with mean 0 and standard deviation $\sigma/\sqrt{n}.$ But that is due to the CLT, not the WLLN.]

m = 5000; x = rexp(m, 1/15); n = 1:m; a = cumsum(x)/n
plot(n, a, type="l", lwd=2, col="blue", ylim=c(5,25))
abline(h = 15, col="green3")

Uniform data: Let $X_1, X_2, ..., X_{5000}$ be independent and distributed as $\mathsf{Unif}(75,125), \sigma = 50/\sqrt{12} \approx 14.43.$ Look at a graph of $A_n = \bar X_n = \frac 1 n \sum_{i=1}^n X_i$ against $n.$ [Even for moderate $n,\, A_n - \mu$ is approximately normal with mean 0 and standard deviation $\sigma/\sqrt{n}.$]

m = 5000; x = runif(m, 75, 125); n = 1:m; a = cumsum(x)/n
plot(n, a, type="l", lwd=2, col="blue", ylim=c(90,110))
abline(h = 100, col="green3")