I'm familiar with information and coding theory, and do know that the units of Shannon information content (-log_2(P(A))) are "bits". Where "bit" is a "binary digit", or a "storage device that has two stable states".
But, can someone rigorously prove that the units are actually "bits"? Or we should only accept it as a definition and then justify it with coding examples.
Thanks!
On
A quick answer could be that if your storage device has probability $1/2$ of being in either of the two possible states, then the formula $$\sum_i -p_i\log_2(p_i)$$ gives you $=1$, so that is the unit. Somehow it sounds like you are not satisfied with that argument given that you have surely seen it?? Note that this does require the two states to be equiprobable. Note also that some (older?) sources occasionally measure information in nats instead of bits. The difference being that the natural logarithm is used in place of base-2 logarithm.
When there are two events, both of which are equally likely, when conveying the news that one of them has actually happened, you convey -log2(0.5)=1 bit of information.
There is no rigorous proof here: just a mapping from a probability space to a bit-space (if I may call so). Either you see a binary random variable with equally likely 0 or 1, or you consider two equally likely stable states, whose representation would entail 1 bit.