How do I estimate the failure rate of a process with limited data?

Question

I am analyzing the pass rate of inspections in factories, but I don't understand how to properly quantify the confidence in my results. For example, if I have a factory that has passed 4 out of 4 inspections - the observed pass rate is 100%, but obviously I have pretty low confidence in that being the 'true pass rate' of the factory.

Some more detail: I am analyzing many factories at once, so I do have an overall mean pass rate and associated variance. But, I have no idea what the parameters of a particular factory would be, all I have is the number of inspections and the observed pass rate over those inspections. Qualitatively, the variance is also quite large - factory quality and the associated pass rate varies wildly.

What I would like, ideally, is to be able to state: I am x% confident that factory A has a 'true pass rate' above y%.

Answer 1

So far this seems unstructured and vague, so I'm not sure I understand either the data or the objectives. You say you may be interested in a Bayesian approach, so let me give a specific example of a Bayesian probability interval for a 'pass rate'.

Prior distribution, The pass rate has a value between $0$ and $1.$ For a Bayesian approach you might use a beta distribution as the prior distribution. Beta distributions have support $(0, 1).$

Maybe the company has been in business for a while and has a reasonably good reputation. Your prior opinion before seeing any data is that the pass rate for this particular company is more likely near 1 than 0. Specifically, you might choose the distribution $\mathsf{Beta}(13, 2)$ to roughly reflect your prior opinion of its pass rate. This distribution has mean $\mu \approx 0.87,$ median $\eta \approx 0.88,$ and puts about probability $0.8$ above $0.8.$ The kernel of this distribution is $p(\theta) \propto \theta^{13-1}(1 - \theta)^{2-1},$ for $0 < x < 1,$ where $\theta$ represents the pass rate. (The proportionality symbol $\propto$ indicates that we have omitted the constant factor that makes the density function integrate to unity.)

13/(13+2)
## 0.8666667          # mean
qbeta(.5, 13, 2)
## 0.8829779          # median
qbeta(.2, 13, 2)
## 0.8008312          # 20th percentile is 0.8

The figure below shows the density function of $\mathsf{Beta}(13, 2)$ with vertical lines at the median (red) and the 20th percentile (dashed orange).

curve(dbeta(x, 13, 2), 0, 1, col="blue", lwd=2, ylab="PDF", xlab="Pass Rate", 
      main="Density of BETA(13,2)")
  abline(v = 0, col="green2");  abline(h = 0, col="green2")
  abline(v = qbeta(.5,13,2), col="red")
  abline(v = qbeta(.2,13,2), col="orange", lty = "dashed")

Data. Now suppose you have data showing that this company passed 93 out of 100 inspections (a 93% pass rate). The binomial likelihood function for these data is $p(x | \theta) \propto \theta^{93}(1 - \theta)^7.$$

Posterior distribution. The posterior distribution is the product of the prior distribution and the likelihood function:

$$p(\theta|x) \propto p(\theta) \times p(x|\theta) \propto \theta^{13-1}(1 - \theta)^{2-1} \times \theta^{93}(1 - \theta)^7 \propto \theta^{106-1}(1-\theta)^{9-1}.$$

We recognize the right-hand side as the kernel of the distribution $\mathsf{Beta}(106, 9),$ which has a 95th percentile of about 96%.

qbeta(.95, 106, 9)
## 0.9581977

The posterior distribution reflects the information in the prior distribution and the information in the data. If you had no prior knowledge of the company you might use a flat or 'noninformative' prior distribution such as $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1).$

Notes:

(1) Finding the posterior distribution was simple here, because the prior distribution and likelihood function are conjugate (mathematically compatible). [This makes it possible to recognize the posterior distribution without having to compute an integral in the denominator of Bayes' Theorem.]

(2) By contrast, if you know nothing about the company in advance, your prior might be $\mathsf{Beta}(1,1).$ And if you had very little data, perhaps 4 passes out of 4, as speculated in your Question, then the posterior distribution is $\mathsf{Beta}(5,5),$ which has 95th percentile about $\theta = 0.75.$