I was reading the following question about the measure of information.
and in it mentioned that $l_i = log \frac{1}{p_{X}(i)}$ is the solution to "Shannon's source compression problem."
I have never heard of such a problem, but I am sure its well known. I was trying to google it but was having a hard time finding some text explaining it. I was wondering if the community new good source about learning about it, either a webpage, a book, lecture notes, etc.
What I am really interested is how the measure of information derived from a optimization problem and as many details as I can get my hands on.
Whoever is voting to close because is to broad clearly did not understand my question. My question is specifically aimed at any reliable source that explains the exact optimization question that I am asking about, nothing more, nothing less. Maybe because you didn't click the reference question I gave you thought I was asking about general information theory. Let me provide the exact problem I want to understand in more detail:
We know (the reference I am looking for) that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$.
This is exactly what I am interested in learning more about, not any general information theory. I am interested in seeing the derivation for the solution, the context of why this problem came up in the first place when Shannon posed it, etc etc. Basically, some reference explaining as much about it as possible. Personally I am mostly interested in the mathematical derivation and formalism of deriving $l_i=\log \frac{1}{p_i}$, but any source that explains something about it is better than nothing.
Also I am not that familiar to information theory, if you provide me references with exact pages and stuff like that, it might be more useful, so that I know exactly at what theorem, idea, chapter or whatever I should be reading. Otherwise, its just way to much to know what I am looking for in an unknown field...