Question is as in the title:
What are some of the first examples of Entropy in a course on Introduction to Probability?
I am TA for a course on Introduction to Probability. I was asked to write down some notes on Entropy of a random variable.
I could not find a reference for a book on Probability that introduces Entropy of a random variable but this is what it is :
Let $(\Omega,\rho)$ be a probability space. Let $X:\Omega\rightarrow \mathbb{R}$ be a discrete random variable with $X(\Omega)=\{a_1,\cdots,a_n\}$ and $P(X=a_i)=p_i$ for $1\leq i\leq k$. Then, \textit{entropy of the random variable $X$} is defined to be the sum $$H(X)=-\sum_{i=1}^k p_i \log (p_i).$$
I can then recall examples of distributions and compute their Entropy to get some idea of what is going on something like
$H(X)=p\log (1/p)+(1-p)\log (1/(1-p))$ for Bernouli distribution $Ber(p)$.
$H(X)= -\log p -q^{2}\log q$ for Geometric distribution $Geom(p)$.
Further, I want to give some other examples that gives a better idea of what role Entropy of a random variable plays. Thus, the question:
What are some of the first examples of Entropy in a course on Introduction to Probability?
If it helps, this course is for second year undergrad (physics/chemistry/biology) students.
The second edition of Achim Klenke, Probability Theory, Springer has a section on "Entropy and Source Coding Theorem" and it contains also the definition. (The book is free at the moment!) Coding theory might be interesting as a non-trivial example.
In my lectures, the first thing we did was to show that the uniform distribution (i.e. $p_i = \frac{1}{k}$) maximises the (mathematical) entropy. This might not be the easiest calculation to start with (it requires constrained minimisation), but it links entropy with a measure of disorder.
Coming from physics, I think it is important to point out the different sign between the disciplines.