How does one determine how much information is provided given number of samples and sample size?

Question

Suppose you have a bag of 10 black and white marbles. What gives you more predictive power: 3 samples of 3 or 12 samples of 1? The numbers are arbitrary of course. I was wondering how one could generalize this - I've tried googling around but couldn't find any answer for this. Could someone point me in the right direction?

Answer 1

I would try to make it concise. First, you need to make a scenario, in which you can see the possibilities. In your problem ordered pairs $\{(x,y)|x+y=10,x,y\in \{0,1,2,3,...\}\}$ give you the possibilities. So, there are $11$ possibilities and we may assume that they are all equally probable. If you want to think in terms of information theory, you should ask: How many bits (how much information)are required to accurately pinpoint one of the possibilities (for example $(3,7)$). It is given by the famous formula

$\log(number of possibilities)=\log11\approx3.46$

Logarithms are in base 2.

So, this is the amount that we need. Now, we need to conduct some experiment to extract some information. Let's do the "12 sample of 1". The first time we draw a ball, it definitely gives us some information. Because it eliminates the possibility of having no ball of that kind. Assume it is white, without the loss of generality. The first draw tells that there is at least one white ball. Then, we calculate the probability of having at least one white ball. It is $\frac{10}{11}$. Using the information theory formula, it gives us

$\log(\frac{1}{p})=\log(\frac{11}{10})$

which is the amount of information, given by the first draw.

For the second draw, either we get the same ball (with probability $\frac{1}{10}$) as the first draw or a new ball (with probability $\frac{9}{10}$). The average information given by the second draw is

$\frac{1}{10}\times(0)+\frac{9}{10}\log(\frac{10}{9})=\frac{9}{10}\log(\frac{10}{9})$

For the next draws, we can apply the same way of thinking to get

$\frac{1}{10}\sum_{x=1}^{10}x\log(\frac{x+1}{x})\approx1.28$

The result sound reasonable, as the amount of information($1.28$), given by the experiment, is less than what is actually required ($3.46$).

Note that you always need to specify what is known and what is to be determined.