How close is the sample mean $\mu_N$ for $N$ random variables to the sample mean $\mu_{N-1}$ for $N-1$ random variables?

Question

Define $\mu_{N}$ to be the sample mean for $N$ i.i.d. random variables that each have mean $\mu$ and variance $\sigma^2$: $$ \mu_{N} := \frac{1}{N} \sum_{i=1}^N X_i. $$

Now consider the sample mean if we removed one of these random variables. That is consider, $\mu_{N-1}$ where w.l.o.g. we remove the last random variable from the sum: $$ \mu_{N-1} := \frac{1}{N-1} \sum_{i=1}^{N-1} X_i. $$

I am interested in describing how close $\mu_{N-1}$ is to $\mu_{N}$ in an appropriate sense..it is this 'sense' that I am unsure of as I haven't been working with probability that long.

Subtracting $\mu_{N-1}$ from $\mu_{N}$ we have \begin{align} \mu_{N} - \mu_{N-1} &= \frac{1}{N} \sum_{i=1}^N X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i \\ & = \frac{1}{N}X_N + \frac{N-1}{N}\frac{1}{N-1} \sum_{i=1}^{N-1} X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i \\ & = \frac{1}{N}X_N + \bigg(\frac{N-1}{N}-1\bigg)\frac{1}{N-1} \sum_{i=1}^{N-1} X_i. \end{align}

So as $N\to \infty$, heurstically it 'looks like' both terms go to zero. But as the $X_i$ are random variables I imagine we need to set some conditions on them for this to hold. For example, we could be unlucky and draw $X_N$'s that are sufficiently large such that $\frac{1}{N} X_N$ does not go to zero as $N \to \infty$.

I feel like the variance $\sigma^2$ could have an effect on the convergence but I'm not sure.

So my questions are

• In the finite sample case can we say something more informative than what I have already derived above regarding how close $\mu_N$ is to $\mu_{N-1}$?
• What conditions are needed on the $X_i$ to ensure $\mu_{N} - \mu_{N-1}$ converges to zero as $N \to \infty$?
• What is an appropriate sense of convergence for $\mu_{N} - \mu_{N-1}$? I am thinking maybe we can say it converges with 'high probability'. Most importantly, is it possible to find an exact rate of convergence?

Answer 1

You need to read about the different concepts of convergence of random variables.

In this case, calling $Z_n = \frac{1}{N} \sum_{i=1}^N X_i - \frac{1}{N-1} \sum_{i=1}^{N-1} X_i$, you want to know if and (in what sense ) $Z_n$ converges to zero. Notice that $Z_n$ is a random variable.

Given that $X_i$ are iid with finite mean and variance, you can deduce that $E[Z_n]=0$ and $\sigma_{Z_n}^2 = \frac{1}{n(n+1)}\sigma_{X}^2$.

This says that $\sigma_{Z_n}^2 \to 0$ (and quite quick).

Then you can apply Chebyshev's_inequality : for any $\epsilon >0$

$$P(|Z_n| > \epsilon) \le \frac{1}{\epsilon}\sigma_{Z_n}^2=\frac{1}{\epsilon \, n(n+1)}\sigma_{X}^2 < \frac{\sigma_{X}^2}{\epsilon \, n^2} $$

Then $P(|Z_n| > \epsilon) \to 0$ as $n\to \infty$

The above amounts to prove convergence in probability from convergence in $L^2$-norm. These are quite strong measures of convergence for random variables, though not so strong as almost sure convergence.

Answer 2

$Z_n\to 0, n\to\infty,$ almost surely iff $\mathrm{E}[|X_1|]<\infty$.

Proof. Sufficiency: by the Kolmogorov SLLN, $\mu_n \to 0$ and $\mu_{n-1} \to 0$, $n\to\infty$, almost surely.

The necessity is proved similarly to the proof of SLLN: since $\mu_{n-1}$ and $X_n$ are independent, it is clear that the both terms in the expression you wrote for $\mu_n - \mu_{n-1}$ must converge to $0$, in particular, $X_n/n \to 0, n\to\infty$, a.s. Then, with probability $1$, $|X_n|/n > 1$ only finitely many times. By the reverse Borel–Cantelli lemma, we then must have $$ \sum_{n=1}^\infty \mathrm P(|X_n|>n) = \sum_{n=1}^\infty \mathrm P(|X_1|>n)<\infty, $$ which is equivalent to integrability of $X_1$.

Using the strong law of large numbers for non-integrable random variables (I can elaborate if you wish), it is also possible to prove that $Z_n \to 0$ in probability provided that $\mathrm{E} [|X_1|^{1/2}]<\infty$, though I have a doubt that this is also a necessary condition (it is necessary for the second term in your expression to vanish almost surely; it is also possible to formulate a necessary and sufficient conditions in terms of behavior of characteristic function at zero).