Right now I'm confused on how Information is even different than entropy.
But let's say we got an information about the result of a coin toss. How much information would that give us? In bits?
I straight up went to calculating Entropy and got 1 bit, which is easily backed up by logic, since 1 bit tells us that a value is either 1 or 0(head or tails).
How about information of a die toss result?
I'm guessing it should carry a lot more information. Using the same process as before i got 6* (1/6 * log2(1/6)) which leaves me with 2.58 bits of information. This could be right but I'm confused. Am I not just calculating regular entropy?
If by some miracle my thinking is correct for a die toss result, how much information would the result of a two-die toss result bring us? My logic would be since each die is bringing us 2.58bits of information, the sum of these should tell us how much information in an two-die-toss. What do you think?
Let's take it a bit further. If we're just randomly guessing 5 numbers out of 50, how much information would the result of that one bring us?
Yes. Actually, entropy is the average information.
In your examples, all events (coin and dice) are equiprobable, so each ocurrence carries the same information. But that's not necessary. Asumme you have a biased coin, with $p(Head)=0.9 $ and $p(Tail)=0.1$. In that case, the information you get each time a Head occurs is $-\log(0.9)=0.152$ bits; each time a Tail occurs you get an information $-\log(0.1)=3.322$ bits (less probable events are "more informative"). In average you have $0.152 \times 0.9 + 3.322 \times 0.1=0.469$ bits - which is the entropy of the coin.