How can I randomly generate a probability mass function such that the entropy of a random variable that follows that probability mass function is a specific value $h$?
Basically, I need to randomly generate a probability distribution over $n$ states with probability $p_i$ of being in state $i$. We can represent the distribution as a vector: $\vec{p}={p_1,p_2,p_3,...p_n}$ such that:
$$\sum_{i=1}^n p_i\ =\ 1$$
and
$$Entropy(\vec{p})=-\sum_{i=1}^n p_i\ log_2(p_i)\ =\ h$$
for a known $h$ between $0$ and $log_2(n)$. Specifically, I know $h$ and I want to randomly select $n$ values that satisfy the above conditions for that $h$.
The set of values ${p_1,p_2,p_3,...p_n}$ should be uniformly distributed. In other words, the probability density function of a point should match the probability density function of a point uniformly distributed on a simplex. See this post about randomly generating points on a simplex. To put it in formal terms, the distribution should be uniform in the sense that it will be the same as though the point were generated through rejection sampling of points on the simplex where any point $\vec{p}$ is rejected if outside of the $Entropy(\vec{p})$ is outside the range range $[h-\epsilon,h+\epsilon]$ for an infinitesimally small $\epsilon$.
Update: If we raise each element of $\vec p$ to a power $x$ and then normalize the result by scaling it so that it sums to one, then we can adjust the entropy arbitrarily (provided that $\vec p$ doesn't represent an exactly uniform distribution). Based on this, I think we can randomly generate a point on the simplex and then just raise it to a power to adjust the entropy to the desired value. Does anyone know how to prove that this won't lead to a bias when compared to rejection sampling?
This is not an answer, I am just answering the Update part with a numerical visualization.
Observation: all the points that can be acheived by a certain starting point are on a curve and they can map to each other with the same method.
Now fixing n=3 for easier visualization I plot $p_1$ and $p_2$ against each other and add the contours of constant entropy:
Now lets assume this method gives us a uniform distribution for $H=1.4$, we can see visually that mapping this into $H=1.2$ will not give us a uniform.
Theoretically such a function should be orthogonal to $(n-1)$D constant entropy surfaces (it's a necessary condition). And on top of that you have to show that your initial probablity density is mapped to a uniform distribuion on a single surface as well (sufficient condition).