What is an intuitive interpretation of the concept of maximum entropy?
I'm confused about what it measures, how it does that (roughly), and where one would apply it. Why do we talk of maximum entropy related to a Gaussian distribution vs, say, related to a uniform distribution? Are there different uses for this term (physics vs statistics)?
Typically, we have some constraints on a distribution (for example, the marginal distributions), and we seek a max-entropy distribution satisfying these constraints. This basically means that, apart from the known constraints, we know as little as possible about the distribution. We do not want to specify anything about the distribution that is not already implied by the known constraints.