Good books on mathematics of information?

Question

Are there any good books on the mathematics of information, under whose umbrella would fall cryptography, information storing, data processing (i.e. data to information), mathematics of artificial intelligence. I understand that many of these topics have originated in computer science, but I want resources on the mathematical foundations of these topics.

Maybe even some mathematics of information as related to quantum computing.

I've also read a little bit of Marcus Hutter's paper 'AIXI' aka 'Universal Artificial Intelligence'.

Answer 1

I think the following make a good start:

1. Cover and Thomas, Elements of Information Theory

This is at an intermediate mathematical level, but has quite a broad coverage, including more chapters on network information theory in later editions (e.g., ed. 3)

2. R G Gallager, Information Theory and Reliable Communication

A classic, very mathematical, and still very enjoyable to read. Deeper than Cover and Thomas but narrower. However it does cover some algebraic coding theory which TCover and Thomas doesn't.

3. D MacKay, Information Theory, Inference and Learning Algorithms

Quite different than the two above, this focuses more on implementations, and links to learning theory and inference. First to cover LDPC codes in detail [which were invented by Gallager in his PhD thesis in the early 60s]

4. R E Blahut, Theory of Remote Image Formation
5. R E Blahut, Algebraic Codes for Data Transmission
6. R E Blahut, Cryptography and Secure Communication
7. R E Blahut, Principles and Practice of Information Theory

Blahut's books form a nice ensemble, mathematically similar to Cover and Thomas.

8. R Vershynin, High Dimensional Probability

Good for statistics/probability of large data sets.

9. D G Luenberger, Information Science

Gentler level of mathematics, interesting broad coverage across different applications of information.