Confidence interval calculation

Question

I need to compute models, which return the probability to reach the goal state per every sample. For some models I have the probability sequence equal to $0,0,0,0,0,0,0,0,0\dots$ and $1,1,1,1,1,1,1,1,1,1,1\dots$

I need to compute confidence intervals for these extreme conditions as well as for different cases, where I have some zeros and ones in the sequence.

I can't use the general formula $$CI = (\mu-z*\sigma/\sqrt{n}, \mu+z*\sigma/\sqrt{n}).$$ here, as well as $t$-distribution formula, because it will return the interval equal to $[\mu-0, \mu+0]$ immediately after the first sample.

I will really appreciate your advice!

Answer 1

You are using the so-called Wald interval, based on normal approximation and other asymptotics. It is OK for huge $n$ (e.g., nationwide public opinion polls). But it is known to have problems (including the one you mention) for small $n.$

Agresti-Couil Interval: Let $\tilde n = n+ 4$ and estimate $\tilde p = \frac{x+2}{\tilde n}.$ Then a 95% CI for $p$ is of the form $$\tilde p \pm 1.96\sqrt{\frac{\tilde p(1- \tilde p)}{\tilde n}}.$$ This amounts to appending two successes and two failures to the actual data before computing the CI.

For $x =0$ successes and $n = 10$ trials, this gives $(-.04,.33),$ which you can interpret as $(0,.33).$ [Similarly for upper endpoints above 1.]

The Agresti interval closely approximates the more accurate (but messier) Wilson interval. The Wilson interval 'inverts the test' for $H_0: p = p_0$ vs $H_a: p \ne p_0,$ rejecting when $|Z| \ge 1.96,$ where $Z = \frac{x/n - p_0}{\sqrt{p_1(1-p_0)/n}}.$ Inverting the test means that the CI consists of 'Acceptable $p_0$'. See Wikipedia for the formula.

Bayesian Interval from Uniform Prior. Using $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1)$ as prior and the binomial likelihood $p^x(1-p)^{n-x},$ the Bayesian posterior is $\mathsf{Beta}(x+1, n-x + 1)$ and the 95% posterior probability interval uses quantiles .025 and .975 of the posterior distribution. For $x = 0$ and $n = 10,$ this interval is $(.002, .282).$ The Bayesian interpretation differs in philosophy, but the numerical results are sometimes used as a confidence interval by frequentists. This interval can never produce endpoints outside of $(0,1).$

> qbeta(c(.025,.974), 1, 11)
[1] 0.002298972 0.282359950

Of course, the Bayesian approach has the advantage of being able to take substantive prior information, if available, into account by using an 'informative' prior.