There is a lot of theory on maximum entropy distributions because they tend to be useful, and can usually be formulated as a convex optimization. My question is, is there any theory for finding a minimum entropy distribution given some constriants (for discrete R.Vs)? In particuar I am interestind in the following problem.
If there is some discrete random vairable $X \sim p_{X}(x)$ which takes on values from $x \in \mathcal{X}$ such that $p_{X}(x) \leq c$ where $0 < c < 1$. Is it possible to find an analytic (or numeric) solution to the optimal distribution $p_{X}^*(x)$ which minimizes:
$$\min_{p_X(x)} H{(X)}$$ $$\text{s.t.}\;\;\; p_X(x) \leq c \;\;\forall x \in \mathcal{X}$$
Consider two elements $x_1,x_2\in\mathcal{X}$ and minimize their contribution to the entropy, given a fixed sum $s$ of their probabilities. So with $q=p_X(x_1)$ and $s=p_X(x_1)+p_X(x_2)$ we want
$$-q\log q-(s-q)\log(s-q)\to\min\;.$$
If neither probability is at one of the boundaries $0$ or $c$, then the derivative with respect to $q$ for fixed $s$ must be $0$ at the minimum. This yields $\log q=\log(s-q)$ and thus $q=s-q$. But that’s a maximum. Thus, this pairwise contribution is minimized when at least one of the two elements has probability $0$ or $c$. It follows that at the global minimum at most one element has any other probability. Thus there are $\left\lfloor\frac1c\right\rfloor$ elements with probability $c$, one element with probability $1-c\left\lfloor\frac1c\right\rfloor$, and all other elements have probability $0$.
P.S.: I just realized that you required that the probability be $\lt c$, not $\le c$. Under this constraint, there is no minimum entropy distribution, but you can get arbitrarily close to the infimum of the entropy by substituting $c-\epsilon$ for $c$ above.