A good textbook to learn about entropy and information theory

Question

I’m wondering if someone could recommend me a good graduate level textbook or resource to better learn about entropy and information theory, divergence criteria and so on. I have had two graduate courses on probability, one on measure theory and one on stochastic control. I’ve only seen very little about this in probability books, and in large deviations books, with a focus on rate functions.

Thanks!

Answer 1

I recommend Patrick Billingsley’s beautiful book “Ergodic theory and information”. Rigorous but very ‘readerfriendly’. The treatment of the concept of entropy is measure theoretical, not topological. It is published in 1965, so you should probably consult a university library. Btw, for both treatments of entropy and their interrelationships, you should consult Walters ‘Ergodic Theory’ (GTM). Also excellent but a bit more demanding.

Answer 2

Try Khinchin's book.

For a more intuitive, thorough, and literary artful discussion of what entropy is, see if you can't get a copy of Renyi's Diary on Information Theory.

Answer 3

I recommend Cover and Thomas,

https://www.goodreads.com/book/show/433439.Elements_of_Information_Theory

as well as Gallager

https://www.amazon.com/Information-Theory-Reliable-Robert-Gallager/dp/0471290483

Answer 4

Late to the party, but for people who will find this answer in the future, I would wholeheartedly recommend the first part of McKay's Information theory, inference and learning algorithms. It is freely available online, and there are accompanying lectures on YouTube.

http://www.inference.org.uk/mackay/itila/