Entropy of a Fair Coin Toss

Question

The problem: A fair coin is tossed until a a heads is reached for the first time. What is the entropy $H\left(x\right)$ in bits?

My solution:

Because $P\{X=i\}=\left(\frac{1}{2}\right)^i$,

$H\left(x\right)= -\sum_{i}^{ \infty}p(x)\log_{2}\left(p(x)\right)$

$\qquad = -\sum_{i}^{ \infty}\left(\frac{1}{2}\right)^i\log_{2}\left(\left(\frac{1}{2}\right)^i\right)$

$\qquad=-\sum_{i}^{\infty}i\left(\frac{1}{2}\right)^i\log_{2}\left(\frac{1}{2}\right)$

$\qquad = \frac{\frac{1}{2}}{\left(1-\frac{1}{2}\right)^{2}}$

$\qquad = 2$.

So 2 bits.

However, the text provides this solution:

Setting $p=\frac{1}{2}$ where $P\left(X=n\right)=pq^{n-1},n \in\{1,2,...\}$ the entropy is

$H\left(X\right)=-\sum_{n=1}^{\infty}pq^{n-1}\log_{2}\left(pq^{n-1}\right)$

$\qquad=-\left[\sum_{n=0}^{\infty}pq^n\log_2\left(p\right)+\sum_{n=0}^{\infty}npq^{n}\log_{2}\left(q\right) \right]$

$\qquad =\frac{-p\log_{2}\left(p\right)}{1-q}-\frac{pq log_{2}\left(q\right)}{p^{2}}$

$\qquad=\frac{-p\log_{2}\left(p\right)-q\log_{2}\left(q\right)}{p}$

$\qquad = \frac{H\left(p\right)}{p}$ and setting $p=\frac{1}{2}$, we have 2 bits.

My questions:

Is my solution valid? Also, if someone could explain in detail how the text provided solution went from this step

$-\sum_{n=1}^{\infty}pq^{n-1}\log_{2}\left(pq^{n-1}\right)$ to this step $-\left[\sum_{n=0}^{\infty}pq^n\log_2\left(p\right)+\sum_{n=0}^{\infty}npq^{n}\log_{2}\left(q\right) \right]$,

as I am a little lost at how to bridge these two parts of the solution. Thanks in advance.

Answer 1

You have the right idea, they just worked in a more abstract setting. Here $p$ is your probability of hitting heads on the next flip and $q$ is the probability of flipping tails. Since there are only two outcomes on a given flip, then $q=1-p$. In this particular case $p=\frac{1}{2}$ so $q=1-p=\frac{1}{2}$. So the probability they give is $$P(X=n)=pq^{n-1}=\Big(\frac{1}{2}\Big)\Big(\frac{1}{2}\Big)^{n-1}=\Big(\frac{1}{2}\Big)^{n} $$ Which is exactly what you found for this particular case.

The other step you mentioned is using the properties of $\log$ and then reindexes the series.

1. $\log(ab)=\log(a)+\log(b)$
2. $\log(a^n)=n\log(a)$
3. $\sum_{n=1}^\infty a_{n-1}=\sum_{n=0}^\infty a_n$