Estimating population standard deviation with sample standard deviation

Question

At 4:30 of this video the author decided to estimate the standard deviation of the population with sample standard deviation (sample size was $100$).

In the next video, the author mentioned that it was reasonable because the sample size greater than $30$. Well, what tells us that we could estimate standard deviation in this way? Why is $30$ that magical boundary? Does it have anything to do with Central Limit Theorem? (I guess not, because we don't calculating the standard deviation of the mean, so it's not related in any way).

Answer 1

Neither of the two methods of estimating the population standard deviation from the sample produces an unbiased estimate, though the $\frac{1}{n-1}$ method does produce an unbiased estimate of the variance.

If you compare the two estimates of the variance $$s_s^2 = \frac{\sum_i^n (x_i - \bar{x})^2}{n-1}$$ with $$s_p^2 = \frac{\sum_i^n (x_i - \bar{x})^2}{n}$$ then clearly $\frac{s_p^2}{s_s^2} = \frac{n-1}{n}$ and so $$\dfrac{s_p}{s_s} = \sqrt{1-\frac{1}{n}} \approx 1 - \frac{1}{2n}$$ which gets closer to $1$ as $n$ increases (for $n=30$ it is about $0.983$ and for $n=100$ about $0.995$) and this factor is less important than the uncertainty in estimating the population standard deviation from a random sample.

Answer 2

At the root, the issue here seems to be whether to use the z-statistic or the t-statistic in finding a confidence interval for the population mean $\mu$ or in testing a hypothesis about $\mu.$

Suppose $X_1, X_2, \dots, X_n$ is a random sample from a normal population of which both the mean $\mu$ and the standard deviation $\sigma$ are unknown. We wish to find a 95% confidence interval (CI) for $\mu.$

If we knew $\sigma$ then $$Z = \frac{\bar X - \mu}{\sigma/\sqrt{n}} \sim Norm(0, 1).$$ Thus $$P\left\{-1/96 \le \frac{\bar X - \mu}{\sigma/\sqrt{n}} \le 1.96\right\} = 0.95,$$ in which $\mu$ can be isolated in a few steps of algebra to $$P\{\bar X - 1.96\sigma/\sqrt{n} \le \mu \le \bar X + 1.96\sigma/\sqrt{n}\} = 0.95.$$ And so we say that a 95% CI for $\mu$ is $\bar X \pm 1.96\sigma/\sqrt{n},$ in which all quantities $\bar X, \sigma,$ and $n$ are known. The numbers $\pm 1.96$ are chosen because they cut 2.5% probability from the upper and lower tails of the standard normal distribution, leaving 95% in the center.

In case $\sigma$ is unknown, it is convenient to use the sample standard deviation $S$ instead, claiming that $\bar X \pm 1.96 S/\sqrt{n}$ or perhaps $\bar X \pm 2 S/\sqrt{n},$ is an approximate 95% CI for $\mu.$ If $n \ge 30,$ this approximation is pretty good, for reasons we see just below.

If $\sigma$ is not known, the exact distribution is
$$T = \frac{\bar X - \mu}{S/\sqrt{n}} \sim T(n-1),$$ Student's t distribution with $n-1$ degrees of freedom. Then an exact 95% CI for $\mu$ is $\bar X \pm t^* S/\sqrt{n},$ where $t^*$ cuts 2.5% of probability from the upper tail of $T(n-1)$ and, by symmetry, $-t^*$ cuts 2.5% from the lower tail. Looking at tables of the t distribution we see that for $n \ge 30$ (or $n-1\le 29$), $t^*$ is approximately 2.0. So the approximate procedure with the standard normal distribution and the exact procedure with Student's t distribution amount to about the same thing.

For smaller values of $n$, the values of $t^*$ get noticeably larger. For example if $n = 10$, we have $t^* = 2.262.$ Thus the 95% CI gets longer (less precise). You can think of this loss of precision as a 'penalty' for having to estimate $\sigma$ by $S$ instead of knowing the exact value of $\sigma.$

There are a few good reasons to forget the 'rule of 30' altogether:

First, it 'works' only for 95% CIs. For a 99% CI we need to cut 0.5% of probability from each tail: the normal cut-off value is $z^* = 2.576$ and we need to increase the sample size to about $n = 60$ before $t^* \approx 2.6.$

Second, in using statistical software, either we know the exact value of $\sigma$ or the program will approximate it from data as $S.$ From the start, we have to know whether we are doing a z-interval or a t-interval. Using an unnecessary rule about sample size only confuses issue. The correct rule is: use z-procedures is $\sigma$ is known (and it usually isn't in practice); use t-procedures of not.

Third, some authors of elementary books try to use the 'rule of 30' (without any theoretical justification) for various kinds of limiting procedures, applicability of the Central Limit Theorem, safe use of t-procedures for non-normal data, and so on. In these applications, 30 is seldom an appropriate dividing line.