Confidence interval - computing task

Question

I'm planning to write a small app for a tedious task:

• Generating pdf based on a existing template (.txt basically)
• Binding specific data (some text, recipient, etc) to that template

Suppose I have 100 pdf to generate. I expect $±100$% confidence if I check manually all the generated pdf.

• How many pdf should I check to reach a level of confidence of $95$%, $99$%, $99,999$%?
• What additional parameter should I deal with?

At first sight, checking only a few could give me a high confidence of reliability. However I'm wondering how to rigorously achieved such confidence using a proper formula.

Thanks for your help.

Answer 1

Here you can find several ways of calculating confidence intervals, wich I'm talking about. https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

I calculate confidence intervals for probability that your program works properly. Here a my results for different types of one-sided 95% confidence intervals, if you check 100 out of 100.

Jeffrey (0.97525, 0.99999)
Agresti-Coull (0.9555, 1.0074)
Direct stepped binomial test (0.96244, 1)
Wilson (0.96300, 1.0)
Beta (0.96378, 1.0)

Edit

These estimates are for the probability that your program works correctly for the random document from the infinite set of possible documents, considering 100 of 100 has been processed correctly.

I've calculated these intervals with python library statmodels using fuction proportion_confint.

If you want confidence intervals for the probability, that your program will work for 100 of 100, having that it has worked for $x$ of $x$, that would be more difficult to calculate. The results for different documents may be highly correlated. Though if we consider them independent, we can go by the following algorithm:

1. choose the confidence level $c$.
2. calculate $c' = c^{1/(100-x)}$
3. calculate the lower bound $lb$ for probability of success of one calculation with the algorithms mentioned above on the level $c'$.
4. the resulting interval is $(lb^{100-x}, 1)$

For example for $x = 80$ and $c=0.95$, the resulting interval would only be $(0.1846, 1)$. With increasing $c$ and decresing $x$, the $lb$ will only get smaller. Because this is the confidence interval for the probability that all $(100-x)$ of your calculations would be successful.