Does the law of large numbers "work better" for certain types of distributions compared to others?
For example, imagine one country where the distribution of income is normally distributed - and imagine another country (same sized population as the first) in which most people earn almost no money, and there is one trillionaire.
Using the law of large numbers, average incomes calculated from random samples using the first country will tend to reflect the true average income of this country .... but in the second country, random samples will likely not be close to true mean unless the trillionaire citizen is included in the sample, and even if he is included, it might not be close.
Thus, in heavily skewed , irregular and multimodal distributions such as the second country - does the law of large numbers some how work "worse" than the first country?
Although increasing the sample size in both countries will likely produce estimates closer to the real mean income .... I have a feeling that fewer random samples are required if the distribution is less irregular... thus, the law of large numbers indirectly is affected by the type of distribution.
- Is this true?
- And if this is true, what does this phenomena (i.e. sample size/accuracy relationship vs irregularity of distribution) referred to?
- In general, can we relate the "irregularity" of a probability distribution to its "skewness"?
Thanks!
I think the answer you are looking for is...
(1) yes
(2) all of the above and more
(3) sometimes
For the LLN (or CLT) we say that if we have independent and identically distributed (IID) data with finite mean and variance , then the sample mean eventually converges to a normal distribution. My understanding of your question is that you want to know that if the type of distribution effects how fast it converges. For example, suppose that you have data from population #1 that has a specific distribution, and from population #2 with a different distribution. You want to know if it possible that one of these distributions converges faster to a normal distribution then another.
Here are a few examples of distributions which fall under LLN. Consider a sample $x_1, x_2, ..., x_n$, and different possible ways to generate IID data.
(1) Standard normal distribution
(2) Binomial Distribution
(3) Mixture distribution
In all three cases, the sample mean would eventually converge to a normal distribution. We get the LLN by looking at the behavior of the distribution of the sample mean as the sample size $n$ goes to infinity. If we make different assumptions we can different distributional approximations that may be more accurate to what we actually observe in the "real world" (i.e. settings with finite data).
In case #1 we automatically have a normal distribution regardless of the size of $n$. So we actually do not need LLN! In fact, it can be shown that the exact distribution for any sample mean (calculated from a finite sample, $n < \infty$) is normally distributed if and only if the $x_1, \dots, x_n$ is normally distributed.
In case #2 for finite $n$ the exact distribution of $\overline{x}$ is not a normal distribution, instead we have that $n \overline{x} \sim Binomial$. The sample mean converges to a normal distribution only when $n$ goes to infinity.
I included case (3) because of the example you gave. In case (3) we can think of this as mixture distribution where some data comes from source #1 with probability $p$ and the other data comes from source #2 with probability $(1-p)$, and both sources make up the population. The population still is IID with finite mean and variance, so it meets the requirements of LLN, it just has a non-standard distribution. If $p$ was very close to $0$ or $1$ then it would be hard to get a (finite) sample that represents the true distribution of the population because some of the values would be so seldom observed. The sample would likely need to be incredibly large. We actually also see this problem in the Binomial example we have in case 2. When $p$ is close to $0$ or $1$ the sample size typically needs to be incredibly large before the normal distribution approximates the sample mean.
As mentioned above, the LLN describes the distribution of the sample mean when the sample size goes to infinity, which we do not have in practice. Letting the sample size go to infinity is a pretty big assumption. We can make other assumptions that will result in different limiting distributions that can be more accurate. Sometimes under these other assumptions we still result in something pretty close to the normal distribution, and other times we do not. When they do result in something close to the normal distribution we could think of these converging "faster" then the other cases. A rate of convergence is definitely at play.