Short version
According to the law of large numbers, how many samples do I need to take to reach the half-life of the convergence towards the mean?
In other words: How big is a large sample?
Long version
I am having a lot of fun getting back to learning math and I just watched a video about the law of large numbers. In basic words it means that the more random samples you take the closer your average value will be to the median (or expected value).
Now I wasn't taught much statistics in school so please forgive me if I am asking something really stupid, I tend to do that.
Basically my question arises from always hearing or reading "If you have a large enough sample", yet no mention of how much is actually a large sample. I made some random number generators and saw the law in action, but my pick for the sample size was around 10000-100000, which was completely random.
One of the comments on the webpage mentioned that the sample size should be the number of options squared divided by the probability of the least probable side, which does not make much sense to me.
Then I thought: The more samples you take, the closer you get to the expected value, but you can take infinite sample and you never actually reach the expected value.
This reminds me of radioactive decay! Both are random, both converge to something and both never fully stop.
I then tried to simplify my question and maybe give it some clear aim point and came up with the following:
How many samples do I need to take to reach the half-life of the convergence? (I assume that it will depend on the probability of the sides of the die roll, or coin toss?)
In other words: How big is a large sample?
P.S. Not sure if my title sounds too similar to some related but not the same question and not sure how to tag this.
Edit: (notes for myself regarding the answer by Ross Millikan)
Sigma levels ($3\sigma$): http://en.wikipedia.org/wiki/Six_Sigma#Sigma_levels
standard deviation = $\sigma = \sqrt{np(1-p)}$
p = probability = 0.5 for coin toss
n = number of trials
To get the number of tosses of a coin required to find the probability of a coin toss with a possible error of $\pm0.1$ with a $3\sigma$ level of accuracy, I would have to do
$3\sigma = 0.1n$
$0.1n = 3\sqrt{np(1−p)}$
where:
$p = 0.5$ for coin toss
$n = (\frac{3}{0.1})^2p(1-p)$
$n = 225$
so I would have to do 225 coin tosses
where the standard deviation $\sigma$ will be
$\sigma = \sqrt{(225)(0.5)(1-0.5)} = 7.5$
$3\sigma = 0.1n = 0.1(225) = 22.5$
expected number of heads and tails is $np = 225*0.5 = 112.5$
As long as we are within 22.5 of 112.5, then the value for p that we get will be $0.5\pm0.1$
If you sample $n$ items that have probability $p$ of having some characteristic ($\frac 12$ for a coin flip, $\frac 16$ for a specific die roll, maybe $\frac 1{10}$ for being under 6 years old) the expected number is $np$. The standard deviation on that number is $\sqrt{np(1-p)}$. If $p$ is not too large we often ignore the factor $(1-p)$ and say the standard deviation is the square root of the number of observed events. If you are trying to measure $p$ with some accuracy, this gives you a measure of how large $n$ has to be. Say you want $p$ within $0.1$ at the $3\sigma$ level. Then you need $0.1n =3\sqrt{np}, n=900p$. So if we sample $450$ coin tosses, we expect $225$ heads with a standard deviation of $15$. As long as we are within $45$ of $225$ we will assess $p$ to be in the range $0.4$ to $0.6$ Since $p$ was fairly large, the standard deviation is actually closer to $11$, but you get the idea.