Maximum likelihood with Bernoulli trials: what to do if there are no successes?

Question

I'm analyzing the security of a secret sharing scheme. One attempt I'm analyzing is "blind luck". I return a random share and hope that noone notices.

The probability $p$ of someone not noticing will be quite small, around $10^{-14}$ [it's meant to be secure, after all]. I have a theoretical lower bound on $p$, but no decent upper bound.

I'm trying to demonstrate that $p$ must be small experimentally. So I simulated $10^9$ attempts: all of them failed. This means the maximum likelihood estimate for $p$ is $0$, which is not useful.

Q: Given that each Bernoulli trial resulted in failure, and it's impractical for me to run enough trials to obtain successes, how can I proceed to find some meaningful conclusions about $p$?

E.g., perhaps we can say something like we can be 99% sure that $p$ is at most [blah].

Answer 1

Use the rule of three for observing zero events. Your 95% confidence interval will be $[0,3/10^9].$ See here for a derivation:

http://www.pmean.com/01/zeroevents.html

Answer 2

From a frequentist perspective: If you have zero successes, any reasonable model will (and should) give you a point estimate of zero for $p$. However, you may use the one-sided interval based on (Wilson, 1927):

$$ p \ge \frac{\hat{p} + \frac{z_\alpha^2}{n} + z_\alpha \sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z_\alpha^2}{4n^2}}}{1+\frac{z_\alpha^2}{n}} $$

From a Bayesian perspective: Here you can actually obtain a positive estimate for $p$. Generally, for estimating a binomial proportion, the $Beta(a_1,a_2)$ prior is used. If your prior guess for $p$ is $10^{-14}$, I would set $a_1,a_2$ such that the mean of this distribution is around that value. The posterior point estimate for $p$ is given by:
$$ \hat{p}_J = \frac{n}{a_1+a_2+n}\frac{x}{n} + \frac{a_1+a_2}{a_1+a_2+n}\frac{a_1}{a_1+a_2}= \frac{a_1}{a_1+a_2+n} $$ for $x=0$ (which is basically the mean of the posterior Beta distribution).

The coverage probability of the credible set found using this method usually goes to zero for $x=0$ hence you may just manually set the lower point to zero. (Brown et. al 2001) suggest to calculate the upper point is as:
$$ U_J(x) = 1-(\alpha/2)^{1/n} $$ However, I cannot be sure that the point estimate and this upper bound will be consistent (i.e. point estimate smaller than upper bound). I suggest that you use one of them.

Which one to use? Well, depends on how much you believe your prior guess! Since $x=0$, you prior guess will strongly influence your Bayesian point estimate, if you choose to use that rather than the upper bound. On the other hand, the Wilson estimate is slightly biased towards $0.5$. (Brown et. al 2001) says that both the Wilson and Jeffrey's methods have decent coverage probabilities.

Finally, I strongly recommend giving the Brown paper a quick read, as it covers a lot of relevant ground related to the case of $x=0$: http://projecteuclid.org/download/pdf_1/euclid.ss/1009213286

Hope this helps.