I wonder what is the theoretical limit of a statistical inference problem. For example we have a model with many parameters, and we can sample many data points from the model. This can be viewed as a noisy channel: input is parameters, output is data points. Can we somehow(I don't know how) calculate the mutual information of this "channel" and say "Yes! The channel capacity is no more than this number, so theoretically we need at least this many times of samples to estimate the parameters!"
(I do know the Fisher information and CR bound, but this is not from a pure information theory.)