Suppose we roll a die 10000 times, and observe that the mean is 3.8. How many sixes did we roll?
I know that this is an application of Sanov's theorem for finite alphabets - if the sample mean of a sequence of iid random variables, $L_n$ is close to $x$, then the empirical distribution is close to the probability measure $\nu$ that has the minimum relative entropy with respect to the probability measure of the sequence $L_n$.
In this case, my sample space is $A = \lbrace 1, 2, 3, 4, 5,6 \rbrace$, and I have that $\mu (a) = \frac{1}{6} \quad \forall \, a \in A $.
The relative entropy between two probability measures is $$ H(\nu \shortmid \mu) = \sum_{a \in A} \nu(a) \log \frac{\nu(a)}{\mu(a)}$$
So my problem reduces to one of minimising $$\sum_{i=1}^6 \nu(i) \log \left(\frac{\nu(i)}{\frac{1}{6}}\right)$$ subject to $$\sum_{i=1}^6 i\nu(i) = 3.8, \quad \sum_{i=1}^6 \nu(i) = 1$$ I don't know how to proceed with this optimisation problem - I have tried to use Lagrange multipliers, but I get a distribution whose sum is greater than 1.
Any hints on how to proceed?
I'm not sure that I've done this right, but here's my attempt at resolving this:
To optimize \begin{equation} \sum_{i=1} ^6 \nu(i) \log \left(\frac{\nu(i)}{\frac{1}{6}} \right) \end{equation} Constrained with: \begin{eqnarray} \sum_{i=1} ^6 i \nu(i) &=& \mu\\\sum_{i=1} ^6 \nu(i) &=& 1 \end{eqnarray}
Using method of Lagrange Multipliers we may rewrite this as: \begin{eqnarray} \sum_{i=1} ^6 \nu(i) \log \left(\frac{\nu(i)}{\frac{1}{6}} \right)+\alpha( \sum_{i=1} ^6 \nu(i)-1)+\beta(\sum_{i=1} ^6 i \nu(i) - \mu) \end{eqnarray} Yielding: \begin{eqnarray} \log(\nu(i))+\log(6)+1+\alpha+i \beta&=&0 \rightarrow \nu(i)=\tilde{\alpha}e^{-i\beta}\\ \sum_{i=1} ^6 \nu(i) &=& 1 \rightarrow \tilde{\alpha} = \sum_{i=1} ^6 e^{-i \beta}\\ \sum_{i=1} ^6 i \nu(i) &=& \mu \rightarrow \{\beta\in\mathbb{R}:\sum_{i=1} ^6 (i-\mu)e^{-i\beta}=0\} \end{eqnarray} The $\tilde{\alpha}$ condition gives normalization. The $\beta$ condition enforces the average.
Testing for reasonable values $\mu=3.5$ is solved for $\beta=0$.
Using a numerical solver for $\mu=3.7$ gives $\beta = -0.1035$ and $\tilde{\alpha}=0.114204$. This is expected as a higher than expected average can only come about by making $\beta < 0$.
$\nu(i) = .114205e^{i\cdot 0.1035}$