Shannon Entropy doesn't seem to capture scaling - is there a different metric I should look for?

41 Views Asked by At

If I survey $2$ people (answers can be A or B) and results are: A (given by man) B (given by woman) then my understanding is that the information entropy $H$, measuring the predictive power over a next survey result without any gender info, would be $H=-0.5\log0.5 - 0.5\log0.5 = 1$. But $H=0$ if we take into account the gender info and try to predict the next result. Hence the information gain obtained by knowing the gender of the next surveyed person is $1$.

However, this does not seem to change at all if I survey $200$ people and get $100$ A's (all by men) and $100$ B's (all by women).

How do I capture the increase in certainty that would occur? Is there a way to weigh the entropy or is it a different measure I'm looking for? Also, how problematic is it that we don't know the actual underlying probability distribution?

(This is a much simplified version of what I'm trying to solve but should get the problem across)