Information theory reference: comparison between Mackay to Thomas and Cover

Question

I'm a computational neuroscience student with a background in mathematics. I want to learn information theory over the summer. I am interested in its applications to neuroscience, machine learning, statistics, etc. I cannot decide between Elements of Information Theory by Cover and Thomas and Information Theory, Inference, and Learning Algorithms by David Mackay. I read recommendations saying they are both great. Also I read that while being more theoretically superficial on information theory, Mackay's book is richer in useful applications, e.g. bayesian inference.

Has anyone read them both? What would you recommend?

Thanks.

Answer 1

They are both great books, but completely different approaches. MacKay's book is full of exercises, visual explanations... it tries to convince you of the intuition of concepts first; Cover and Thomas is less visual, less intuitive, more "mathematical".

Personally, I found the narrative of Cover and Thomas hard to grasp, while MacKay is more well organized and explains to you not only the concepts but why we need them. Conversely, for some reason, I kept going back to Cover and Thomas to understand a specific concept or proof.