Is the usage of unbiased estimator appropriate?

Question

Sometimes I find the usage of unbiased estimator quite confusing. For example, the unbiased estimator of variance:$$S^2=\frac{\sum (X_i-\bar{X})^2}{n-1}\,.$$

True, it is the expectation of variance. But when should we use it? I mean, there are other ways to estimate $\sigma^2$, such as MLE. How can I know when I should use MLE and when I should use unbaised estimator?

Secondly, some books(such as A-level and AP syllabus and textbooks) uses $S$ as an estimation of standard deviation. However, $S$ is NOT an unbiased estimation. This perplexes me a lot because I don't know what they are trying to do. Why don't they use an unbiased estimator of standard deviation instead? Here are some unbiased estimators of standard deviation. https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Background

So I have two questions:

1. When should we use unbiased estimator? How can I choose between Maximum Likelyhood Estimation and Unbiased Estimation?

2. Why in some books, unbiased estimator are used in such a way that it end up with bias?

Answer 1

There is no unanimous criterion. All you need to know is that given certain criterion, you prefer this one to that one, and so.

Unbiased estimators don't assure a good estimate per se. Sometimes between an unbiased estimator with a very large variance and another one with a little bias and a much smaller variance, you will prefer the second one. The MSE criterion, which chooses between several estimators $\hat \theta_k$ of the parameter $\theta$ that with lesser MSE (if there is one), uses that same idea, since $$MSE_\theta(\hat \theta)=E(\theta-\hat\theta)^2$$ and its easy to prove that $$MSE_\theta(\hat \theta)=(Bias(\hat \theta))^2+Var(\hat \theta).$$

But even so, it's not a universal truth that $MSE$ is the measure of accuracy. Why squaring instead of absolute value? Why expectation instead of median...?

The best criterion (if there's such a thing) depends on the real world problem you analize; given that criterion, there might be (or not) a best estimator. The mathematical background is not enough to give an absolute answer.

Answer 2

You've pinpointed an important problem with unbiasedness as a desideratum for an estimator, and that is that it's not invariant under reparameterization. The same thing happens with an exponential distribution. There are two common parameters to use, the rate $\lambda$ or the mean $\theta=1/\lambda.$ MLE is invariant so what you get either way is consistent: $$ \hat\theta_{MLE} = \overline X\\\hat\lambda_{MLE} = \frac{1}{\overline X}$$ where $\overline X$ is the sample mean. However, since generally $\tfrac1{E(X)} \ne E\left(\tfrac1{X}\right),$ it turns out that while $\hat \theta_{MLE}$ is unbiased, $\hat \lambda_{MLE}$ is biased.

An obvious answer would seem to be that we should use the bias-adjusted estimator for whichever version of the parameter we "care about" more, or in other words, which parameter's interpretation is more in line with what we are intuitively trying to measure by estimating. By this standard, one might think we should be using an unbiased estimator for the standard deviation rather than the variance, since the standard deviation is intuitively a size of an average fluctuation.

As straightforward as this sounds, there are a number of problems with this line of thinking. The first is pretty minor but still worth noting: actually, the standard deviation isn't the size of an average fluctuation! That would be something closer to the mean average deviation, and for normal distributions this is different by a factor of $\sqrt{2/\pi}$ (or something like that... don't quote me).

Which brings me to the second more important point. What is the formula even for the bias-adjusted standard deviation? It's very complicated compared to the bias-adjusted variance (for the normal distribution). Furthermore, the unbiased variance estimator has a nice property: it is unbiased regardless of distribution. The precise form of the unbiased estimator for the standard deviation depends on the distribution. So that said, it's pretty obvious why authors prefer the unbiased variance estimator.

(Also the unbiased estimator, is a misnomer. I mean the estimator proportional to the square root of the standard variance estimator with the proportionality constant chosen to make it unbiased.)

Fortunately the authors aren't sacrificing much for the sake of parsimony: unbiasedness is an extremely overrated property and we shouldn't care too much about it. Think about what it means: it means that if you do the experiment where you collect sample size $n$ a million times, the average value you get for the estimator is exactly, squarely equal to the true parameter. Think about this literally: is this actually what you want? It seems like ideally this would probably be the case, but we're missing an important dimension of estimator variance. Surely we'd prefer an estimator whose mean is $1\%$ higher than the true value and whose fluctuations are $2\%$ to one whose mean is exactly the true value and whose fluctuations are $20\%.$

One popular metric for the quality of an estimator is mean squared error. This includes contributions from both variance and bias. And it's generally not equal to the unbiased estimator. However, like the unbiased standard deviation estimator, it depends on the distribution... which, between that and the additional conceptual overhead explains why that one isn't 'standard'.

As for why we typically use the bias corrected variance estimator rather than the MLE, it's really just that usually bias-corrected MLEs have marginally better finite sample efficiency than uncorrected. There's also the fact that the unbiased version is the one that makes the formula for the t-test the least cumbersome, which is an explanation that probably shouldn't be overlooked.

Answer 3

I'll start with a quick answer to you practical problem ant then will elaborate a little on the basic notions. So, if you have relatively small sample size - then you should care about the bias. As such, as long as the unbiased estimator is a simple adjustment of the MLE (or some other "legitimate" estimator), then use it. If you have large sample size, then it does not matter as the MLE's bias vanishes asymptotically and it is neglectable for large sample sizes.

Now, for the more general question - What are the desired properties of a "good" estimator? As already have been said, there is no universal criteria, but I guess that no one will argue that (1) consistency and (2) stability are important features. By consistency we assure that our estimator converges in probability (or almost surely for strong consistency) to the true value of the parameter. Moreover, it assures that your estimator will be asymptotically unbiased, i.e., $$ \lim_{n\to \infty} \mathbb{E}[\hat{\theta}] = \theta. $$ And stability - that you estimator will not fluctuate much. Which brings us to the basic measure of stability - its MSE that incorporates both its variance and again its bias, i.e., $$ \operatorname{MSE}(\hat{\theta}) = \operatorname{Var}(\hat{\theta}) + b^2(\hat{\theta}). $$
As such, once again the bias is involved, as large bias will increase the MSE substantially, and the first feature of consistency applies (given a finite second moment) that $$ \lim_{n\to \infty}\operatorname{MSE}(\hat{\theta}) = 0. $$ I.e., asymptotic unbiasedness is again a consequence of a more basic feature. Furthermore, let me convince you that include the unbiasedness as a primary estimator's property isn't smart as you may end up dealing with inadmissible estimators. Assume that you have $X_1,...,X_n$ i.i.d from $\mathcal{N}(\mu, \sigma^2)$, then I can define silly estimator like $\hat{\mu} = X_1$ that is unbiased, as $$ \mathbb{E}[X_1]=\mu. $$
Howevre, its MSE is $\sigma^2$, and thus it is an iconsistent estimator. In this manner I can construct plenty of useless unbiased estimators. As such, including unbiasdness as a primary desirable feature isn't the right approach. So... why so much ado about this bias thing? Apparently, it can be useful as a secondary desirable property. I.e., consistency that applies asymptotic unbiasedness and vanishing MSE is asymptotic property and I never witnessed an infinitely large or near infinitely large sample size. Hence, when dealing with real world statistics - considering the size of the bias may be valuable. However, there are also useful theoretical considerations. If you want to establish some general theory of evaluating estimators, you will have to reduce or restrict the class that you deal with. In such a case, restricting your attention only to unbiased estimators will lead you to important and insightful ways to obtain and compare estimators - UMVUE, the Gauss-Markov BLUE in regression analysis and more. Namely, once you restrict your class only to unbiased estimators apparently, you can derive general theorems regarding the features and ways of construction (e.g., Lehmann-Scheffe theorem) globally (over this class) optimal estimators. Hence, to sum it up. Bias as itself is not that important. What is more important is consistency and stability, and once you set your goal to achieve these two properties, then by dealing only with unbiased estimators you can derive uniformly best possible unbiased estimators. Namely, being sure that you cannot improve your estimators. But once you leave this realm of only unbiased estimators - you are stumble into the bias-variance tradeoff dilemma (e.g., in regression analysis you can reduce substantially the MSE of the estimators with methods like LASSO and/or Ridge regularizations, but it will introduce bias to your estimators) that opens one again the whole problem of finding optimal estimators.