CLT and summing IID Bernoulli RV vs. CLT and summing other types of IID RV

Question

I was curious...

There are several problems in a intro to probability textbook that ask you to prove that:

1. If IID sequence is of Geometric RVs, then the sum of these random variables is a new random variable with a negative binomial distribution

2. if IID sequence is of Exponential RVs, then the sum of these random variables is a new random variable that has a gamma distribution

3. if IID sequence is of Poisson RVs, then the sum of these random variables is a new random variable that is also a poisson RV.

4. Summing squared IID N(0,1) RV, gives you a RV with a chi-squared distribution...

etc...etc..

What I don't get, is how this fits in with Central Limit Theorem?

CLT: Summing IID RV approaches a Gaussian distribution as sample size approaches infinity.

Isn't this a contradiction to the other proofs of adding other types of RVs and NOT getting a Gaussian distribution? what gives?

Answer 1

What you wrote is not a definition of CLT. Instead, it is $$ \frac{S_n - n \mu}{\sigma \sqrt{n}} \to_n N(0,1) $$ assuming each $X_i$ is integrable and has a finite second moment, and all rvs are iid. This doesn't contradict any of the examples you gave, because they are specific to the distribution of those rvs; CLT applies to all rvs that fulfill the requirements of CLT.

Answer 2

There is no contradiction. What the CLT is telling you is that the 'output' distributions that you've named: (1) the negative binomial, (2) the gamma, (3) the Poisson, (4) the chi-squared, and of course (5) the binomial distribution, all have normal approximations, when the distributions in question arise as the sum of $n$ iid random variables, and after suitable scaling. Moreover, the CLT asserts that the approximation improves with $n$. You should be able to find questions on Math.SE asking about the normal approximation to each of these distributions.

Answer 3

The CLT holds only for (1) $n\to \infty$, and (2) appropriate scaling, while the exact distributions that you stated hold only for (1) finite $n$ and (2) without scaling. Any other variations are mere approximations and not exact results.

Answer 4

Let's take the case of the Poisson RV's as an example of what is going on.

The distribution of a Poisson RV is $$ P_\lambda(k)=\frac{\lambda^ke^{-\lambda}}{k!}\tag1 $$ where $\lambda$ is the expected value of the RV; it is also the variance of the RV. Summing $n$ RVs gives a distribution of $$ P_{n\lambda}(k)=\frac{(n\lambda)^ke^{-n\lambda}}{k!}\tag2 $$ To make the mean $0$, we translate by $n\lambda$: $$ P_{n\lambda}(k+n\lambda)=\frac{(n\lambda)^{k+n\lambda}e^{-n\lambda}}{(k+n\lambda)!}\tag3 $$ Note that as with any convolution of random variables, this produces a thinned out distribution; that is, the distribution gets spread out over a larger range and the probability for any given range is decreased commensurately. To counter the effect of this thinning, we scale back by $\sqrt{n}$ to get a distribution with mean $0$ and variance $\lambda$: $$ \begin{align} \sqrt{n}P_{n\lambda}(\sqrt{n}k+n\lambda) &=\sqrt{n}\frac{(n\lambda)^{\sqrt{n}k+n\lambda}e^{-n\lambda}}{\left(\sqrt{n}k+n\lambda\right)!}\tag4\\[6pt] &\sim\frac{\sqrt{n}}{\sqrt{2\pi}}\frac{(n\lambda)^{\sqrt{n}k+n\lambda}e^{-n\lambda}e^{\sqrt{n}k+n\lambda}}{\left(\sqrt{n}k+n\lambda\right)^{\sqrt{n}k+n\lambda+1/2}}\tag5\\[3pt] &=\frac1{\sqrt{2\pi\lambda}}\frac{(n\lambda)^{\sqrt{n}k+n\lambda+1/2}e^{\sqrt{n}k}}{\left(\sqrt{n}k+n\lambda\right)^{\sqrt{n}k+n\lambda+1/2}}\tag6\\[3pt] &=\frac1{\sqrt{2\pi\lambda}}\frac{e^{\sqrt{n}k}}{\left(\vcenter{1+\frac{k}{\sqrt{n}\lambda}}\right)^{\sqrt{n}k+n\lambda+1/2}}\tag7\\ &\sim\frac1{\sqrt{2\pi\lambda}}e^{-\frac{k^2}{2\lambda}}\tag8 \end{align} $$ Explanation:
$(4)$: substitute $\sqrt{n}k\mapsto k$ in $(3)$ and multiply by $\sqrt{n}$
$(5)$: apply Stirling's Approximation
$(6)$: cancel $e^{n\lambda}$ and $(n\lambda)^{1/2}$
$(7)$: cancel $(n\lambda)^{\sqrt{n}k+n\lambda+1/2}$
$(8)$: $\left(\vcenter{1+\frac{k}{\sqrt{n}\lambda}}\right)^{\sqrt{n}k}\sim e^{\frac{k^2}\lambda}$
$\phantom{\text{(8):}}$ $\left(\vcenter{1+\frac{k}{\sqrt{n}\lambda}}\right)^{n\lambda}\sim e^{\sqrt{n}k-\frac{k^2}{2\lambda}}$
$\phantom{\text{(8):}}$ $\left(\vcenter{1+\frac{k}{\sqrt{n}\lambda}}\right)^{1/2}\sim 1$

Formula $(8)$ is normal distribution with mean $0$ and variance $\lambda$.

---

Here is the Poisson distribution, $P_{n\lambda}$ for $\lambda=1$, scaled and plotted for various $n$. The points are placed so that $\sqrt{n}k+n\lambda\in\mathbb{Z}$.

The continuous curve is the Normal distribution with variance $1$.

Thus, the Poisson distribution, when scaled to counter the thinning, tends to a normal distribution.

At no point is the Poisson distribution a normal distribution. For one thing for any $n$, $P_{n\lambda}$ is a discrete distribution; the normal distribution is a continuous distribution. This is a property of limits: it is not necessary that at any point, a sequence equals its limit.

Answer 5

CLT says something approaches a Gaussian distribution, not that it is a Gaussian distribution.

Suppose $X\sim\operatorname{Poisson}(\lambda).$ Then the distribution of $\dfrac{X-\lambda}{\sqrt\lambda}$ approaches $\operatorname N(0,1)$ as $\lambda\to\infty.$

Suppose $X\sim\operatorname{Gamma}$ of the form $\displaystyle \frac 1 {\Gamma(\alpha)} \left( \frac x \mu \right)^{\alpha-1} e^{-x/\mu} \left( \frac{dx}\mu \right)$ for $x>0.$ Then the expected value is $\alpha\mu$ and the variance is $\alpha\mu^2,$ so $\dfrac{X-\alpha\mu}{\mu\sqrt\alpha}$ approaches $\operatorname N(0,1)$ as $\alpha\to\infty.$ A gamma distribution with a large value of $\alpha$ is approximately a normal distribution with the same expected value and the same variance.