Consider a coin with probability of landing on heads, $p$, where $p$ is sampled from a truncated normal distribution from [0,1].
You are offered to play a game where you can bet on a 12% interval containing $p$ of a coin after observing the outcome of $n$ flips, i.e. if you estimate an interval of [0.6, 0.72] and $p$ is contained in this interval you win the game.
What is your optimal strategy for playing this game? What value of $n$ lets you confidently play this game (assuming a 1:1 payout structure, at what point does your strategy net you a win percentage greater than 50%)?
The answer will of course depend on the standard deviation of the distribution, which for my simulation purposes I have just used a placeholder of $\sigma=0.2$. I am wondering if anyone has thoughts on how to obtain a closed form solution or way of estimating what relations between $n$ and $\sigma$ make it worth playing this game.
Initial Thoughts
A trivial strategy is to have the 12% interval centered about 0.5 as this would be the interval containing the most of the distribution and would have probability of success $P(X\le0.56)-P(X\le0.44)$.
Another strategy I considered is to construct an interval centered about the observed probability of the coin, but with a bias headed toward 0.5. For example, after 4 flips I observe 3 heads, and thus construct an interval of [0.69, 0.81], and then consider adding an offset to my bounds to shift them closer to the center.
I have simulated a scenario with $N(0.5, 0.2)$ to varying the number of observed flips, $n$, and what is the best offset strategy at each $n$. As predicted, the benefit of offsetting the interval becomes less significant as the number of flips you get to observe increases.
From my simulations, it seems that $n=20$ leads is the cutoff for when it becomes worth playing this game. Is there anyway to approximate this result without doing simulation?
Let $\theta$ a candidate value of $p$, and $\Theta$ a random variable denoting its random selection, so that $\Theta \sim \mathcal{TN}(\mu,\sigma^2)$ (a truncated normal distribution).
Suppose you observe the $k$ of the $n$ coin flips are heads. Then if you select the interval centered at $c$, the probability you win is given by
$$\Pr[\text{win}_c] = \Pr[c-0.06 \le \Theta \le c+0.06] = {\int_{\theta=c-0.06}^{c+0.06} {n \choose k} \theta^{k} (1-\theta)^{n-k} \, p_\Theta(\theta) \; \text{d}\theta \over \int_{\theta=0}^{1} {n \choose k} \theta^{k} (1-\theta)^{n-k} \, p_\Theta(\theta) \; \text{d}\theta}.$$
Here, $p_\Theta(\cdot)$ is the pdf of $\Theta$, ${n \choose k} \theta^{k} (1-\theta)^{n-k}$ is the probability of observing $k$ heads if the coin's heads probability is $\theta$, and the above formula is obtained by Bayes rule. Of course, the pdf of the truncated normal is given by $p_\Theta(\theta) = {1 \over Z} f(\theta)$ where $f(\cdot)$ is the pdf of the non-truncated normal, i.e.,
$$f(\theta) = {1 \over \sigma \sqrt{2\pi}} e^{-{1 \over 2} ({\theta-\mu \over \sigma})^2},$$
and $Z$ is an appropriate normalizing constant, specifically, $Z = \int_{\theta=0}^1 f(\theta) \; \text{d}\theta$.
The constant $Z$ will cancel out, so we obtain
$$\Pr[\text{win}_c] = {\int_{\theta=c-0.06}^{c+0.06} {n \choose k} \theta^{k} (1-\theta)^{n-k} \, f(\theta) \; \text{d}\theta \over \int_{\theta=0}^{1} {n \choose k} \theta^{k} (1-\theta)^{n-k} \, f(\theta) \; \text{d}\theta}.$$
So, the optimal strategy is to choose $c$ that maximizes $\Pr[\text{win}_c]$, i.e., that maximizes the ratio of two integrals above. Also, the value of the ratio tells you the probability you will win if you play this strategy, which tells you how confident you can be in playing the game. Since it ultimately depends on $n$, it captures the dependence on $n$, and so you can try different values of $n$ to see the smallest $n$ such that the probability of winning is large enough for you.
I don't know whether there is a closed-form formula for evaluating the above two integrals, but you could try plugging it into Wolfram Alpha to see if it can evaluate these integrals.