Where Gaussian Random Distribution came from?

Question

I'm studying an introduction to probability, and before the explanation of the Central Limit Theorem the author presents the Gaussian Normal Distribution.

It has a complex and a non-intuitive formula, even if it's very important in the Probability field. Was the gaussian distribution found before the deduction of the CLT ? How was this distribution discovered ? And what are the properties that makes it so special ?

Answer 1

Special and characteristic properties:

• If $X$ has normal distribution then so has $aX+b$ where $a,b$ are constants with $a\neq0$.

• If $X$ and $Y$ have joint normal distribution then so has $X+Y$.

Note that - if $X_n$ and $Y_n$ are sequences of random variables tending to Gaussians $X$ and $Y$ when $n\to\infty$ on base of CLT - also $X_n+Y_n$ will tend to a Gausian $Z$, so that $X+Y=Z$ where the LHS is a sum of Gaussians. This kind of explains why a summation of Gaussians is a Gaussian again, and tells us why Gaussian distribution is the only "candidate" for the CLT.

Similarly $aX_n$ will tend to a Gaussian, implying that $aX$ is Gaussian again.

Answer 2

I want to comment on "It has a complex and a non-intuitive formula".

Let us first look at the reduced equation,

$$p(x)\propto e^{-x^2/2}.$$

This is simply the exponential of a square, nothing really difficult. The coefficient $2$ serves as a scaling factor to ensure that the standard deviation is $1$. (I won't give the technical details here as this involves improper integrals.)

Now we need to normalize the function so that the area under the curve is $1$ and the final formula is

$$p(x)=\frac1{\sqrt{2\pi}}e^{-x^2/2}.$$

If we consider general mean $\mu$ and standard deviation $\sigma$, the formula is simply adapted by applying the linear transform $x\to(x-\mu)/\sigma$ that shifts $\mu$ to $0$ and rescales $\sigma$ to $1$. To ensure that the area remains $1$, the normalization factor needs to be multiplied by $\sigma$ to compensate. We have

$$p(x)=\frac1{\sqrt{2\pi}\sigma}e^{-(x-\mu)/2\sigma^2}.$$

If you look at the shape of the function, is very smooth, it is symmetrical, it has a single maximum and quickly decays on both sides. This behavior describes a random variable that remains "concentrated" around the mean.

---

To better understand the shape, you can look at the distributions of the sums of $n$ uniform random variables in the range $0,1$, for increasing $n$:

You are smoothing and smoothing the curve, which quickly converges to a Gaussian.

Now observe that with $x+y=z$, where $x,y$ follow a reduced Gaussian

$$\int_{-\infty}^\infty e^{-x^2/2}e^{-y^2/2}dx=\int_{-\infty}^\infty e^{-x^2/2}e^{-(z-x)^2/2}dx=\int_{-\infty}^\infty e^{-(x-z/2)^2/2-z^2/4}dx \\=e^{-z^2/4}\int_{-\infty}^\infty e^{-x^2/2}dx\propto e^{-z^2/4}.$$

This proves that the distribution of the sum of two Gaussians is another Gaussian, with variance $2$. By a more complicated argument (based on the Fourier transform of convolutions), one can show the reciprocal: an additive distribution must be Gaussian.

Answer 3

Speaking to the historical question, it was "discovered" from an attempt to work out what distribution a sample mean should have, and there were some false starts on the way, but I'll leave it there because this isn't HSM.

The Gaussian distribution isn't that non-intuitive, if you look at it the right way. If the log-pdf has a Taylor series and we expand around the mode (which can be taken as $0$ after a translation), the lowest-order term that can survive is an $x^2/2!$ term. The integral $\int_\mathbb{R}\exp -x^2/2\, \text{d}x$ was obtained in the late 1700s (the proof everyone knows today took a little longer; the early arguments, though valid, weren't as short or clean), and it wasn't much harder to verify that the pdf proportional to $\exp -x^2/2$ is of variance $1$, which is a nice normalisation. The "complex" formula you're thinking of is just a linear generalisation, viz. $f(x)\mapsto \tfrac{1}{\sigma}f(\tfrac{x-\mu }{\sigma})$. This is of course slightly neater if you work with CDFs first, viz. $F(x)\mapsto F(\tfrac{x-\mu }{\sigma})$.

There are several reasons Gaussian distributions can arise naturally, besides the CLT. These are discussed in my and others' answers here. My answer in particular discusses several problems to which a Gaussian is the unique or optimal solution.