Testing the probability of a Bernouilli variable

Question

I have a Bernouilli variable which is $1$ with probability $p$. I need to test the hypothesis $H_0:p<\theta$ vs. $H_1:p>\theta$, where $\theta$ is a given constant. The question is to find $n$ such that the test can conclude $H_0$ with a probability $\alpha$ of error.

I understand that after $n$ drawings, the number of $1$'s, let $m$, follows a Binomial law $B(m;n,p)$; it can be approximated by a Poisson or Normal law under some conditions on $n$ vs. $p$ (my $p$ is small). I also understand that the test will conclude $H_0$ when $\text{cdf}_B(m;n,\theta)<\alpha$.

But my unknown is $n$. How can I choose it, either using the exact model or the approximations ?

This is not a homework but a real-world question (quality control). $\theta=0.001, \alpha=0.9$.

Answer 1

First, let's set the stage for what we mean by error:

• false positive (Type I error): we reject the null hypothesis when it is true.
• false negative (Type II error): we fail to reject the null hypothesis when it is false.

The probability of Type I error is commonly called $\alpha$; the probability of Type II error is commonly called $\beta$. The quantity $1-\beta$ is commonly called statistical power of a test.

We want to test the hypotheses $$\begin{align} H_0: \quad p\leq \theta\\ H_1: \quad g > \theta \end{align}$$

To perform this test we choose a test statistic. As you pointed out, one correct statistic is the number of successes $n$ in $N$ trials; then we set $$ T = \sum_{k=1}^N X_k $$ where each $X_k$ is a Bernoulli trial with an unknown probability $p$.

The null hypothesis is rejected if the observed test statistic is below a critical value $\kappa$: $$ T_{\mathrm{obs}} > \kappa \Longrightarrow \text{ reject }H_0.$$

We can compute the probability of rejecting the null hypothesis ($\alpha$) with $$ \alpha = P(T > \kappa|H_0) = 1 - \mathrm{cdf}_{H_0}(\kappa).$$

Then, we find the critical value $\kappa$ such that a $1-\alpha$ portion of the distribution of the statistic is before $\kappa$ under the null hypothesis: $$\kappa = \mathrm{cdf}_{H_0}^{-1}(1-\alpha).$$

For instance, suppose that $N=2000$ and we want a Type I error rate of $\alpha=0.1$ with $\theta=0.001$, then $$ \kappa = 4.$$

The following julia snippet shows how to compute this value easily:

julia> using Distributions

julia> α = 0.1; N = 2000; θ = 0.001;

julia> κ = quantile(Distributions.Binomial(N, θ), 1-α)
4

If I observe more than $n=4$ successes out of $N=2000$ trials, the null hypothesis can be rejected with $\alpha = 0.1$.

Now to the Type II error. How do we determine the statistical power of this test? The problem here is that the statistical power depends on the true value of the probability $p$, which is unknown to us.

What is commonly done is to present the statistical power of the test in function of the true parameter in a formulation akin to saying that "if the true probability were $p$ instead, the test would fail reject the null hypothesis with probability $\beta$". For this test, the power would look something like the following plot

julia> p = range(θ, stop = 0.1, length=100);

julia> f = [1-cdf(Distributions.Binomial(N,pi),κ) for pi in p];

julia> plot(p,f, label="")

julia> xlabel!("p")

julia> ylabel!("Power")

Now, we can plot this for different values of $N$ and see how powerful our test is for different values of $N$.

One example analysis would be "suppose that in fact the probability was 0.0015, how big should $N$ be in order for the test to realize that $p>\theta$ with 90% probability?". To answer this, we would plot the power for increasing $N$ with the following code

julia> θa = 0.0015; N = 100:10000;

julia> plot(N, Power, label="θa = $θa")

julia> plot!([0,10000], [.9,.9], label="90%")

julia> ylabel!("Power")

julia> xlabel!("N")

This plot shows that, in case the true $p$ is 0.0015, then we need a sample size of about $N=2500$ to have 90% confidence that the method would correctly reject the null hypothesis that $p<0.001$.