A friend of mine sent me a stats question, and since stats is definitely one of my weak points, I struggled a bit with this one, and I'm looking for some help.
The question is
Imagine a production line with $N$ units. Each unit has probability $p$ of being defective, but we don't know $p$. How many units do we have to inspect in order to be 95% confident that there are no defects in the batch?
I can't claim to have a good understanding of what a confidence interval means, but here's my attempt. If we've observed some data $X = x_0$, and we want to estimate the underlying parameter $p$ with confidence $1-\alpha$, then we're looking for the interval in which our observation isn't "unusual"; i.e., $P(X=x_0;p) > \alpha$.
So, since our observation is "we saw no defects in our sample", and we're trying to estimate the number of defects total (which isn't a parameter!), I think a sensible interpretation is:
A number of defects $d$ is in our confidence interval iff $P(\textrm{no defects in sample} | d \textrm{ defects in batch}) > \alpha$
In our case, we want our sample to be large enough that our confidence interval is just $0$. So I think the question I want to answer is:
For what sample size does $P(\textrm{no defects in sample} | \textrm{some failures in the batch}) \le \alpha?$
So here's my approach to that problem, whether or not it's the correct problem.
If $S$ is the event that we have no defects in the sample, and $F$ is the event there are no defects in the batch. Also let $q = 1-p$, and $m$ be our sample size. \begin{align*} P(S | F) &= \frac{P(S) P(F | S)}{P(F)} \\ &= \frac{q^m (1 - q^{N-m})}{1 - q^N} \\ &= \frac{q^m - q^N}{1 - q^N} \end{align*}
So we're looking for $m$ such that $\frac{q^m - q^N}{1 - q^N} \le 0.05$. But we don't know $q$. From a common result, our confidence interval on $p$, given we haven't seen any defects in the sample, is approximately $[0, 3/m]$. Worst case scenario it's $3/m$, so we'll use that: $$ \frac{(1-3/m)^m - (1-3/m)^N}{1 - (1-3/m)^N} \le 0.05$$
From there it's numerical.
Was this at all the right approach? If not, what's a more correct one?
Suppose the alternative to no defect is that there is one defect in a batch of $N$:
Then if you sample $m$ without replacement, the probability of not spotting that defect is $1-\frac{m}{N}$. So to have a probability of $0.05$ or less of failing to find that defect you need $m \ge 0.95 N$, so rounding up
If sampling with replacement, then the sample size $m$ would have to satisfy $\left(1-\frac1N\right)^m \le 0.05$ i.e. $m \ge \frac{\log(0.05)}{\log\left(1-\frac1N\right)}$, which can be approximated using $e^{-m/N}\lesssim 0.05$ requiring $m \gtrsim -N \log_e(0.05) \approx 3N$ corresponding to the rule you seem to have found (empirically, $3N-1$ turns out to be big enough)