Usage of $\hat p$ and $p_0$ in confidence intervals/hypothesis testing

Question

Cheers, I have a question about the usage of $\hat p$ and $p_0$ when talking about the confidence intervals of proportion problems.

My professor and my textbook both state that for Bernoulli trials (So $X_1, X_2, \cdots X_n$, $X_i \sim Bernoulli(p)$), with $n \geq 30$, then $$Z = \frac{\hat p - p_0}{\sqrt{\frac{\hat p (1- \hat p)}{n}}}$$, and $Z \sim N(0,1)$

However, I came across a webpage, and some tutorials which instead of the above type, they used the following one: $$Z = \frac{\hat p - p_0}{\sqrt{\frac{ p_0 (1- p_0)}{n}}}$$

Why is that change there on the variance? Does it have to do with the sample size, or is it something I am omitting or understanding wrong? Thanks a lot.

EDIT: An answer is given in the note here, but I would like a more intuitive explanation if possible =)

Answer 1

Before answering your question, let me clear a few things up. I assume that $\hat{p}$ is the empirical estimator for the unknown parameter $p$, that is $\hat{p} = \frac{1}{n} \cdot \sum_{i=1}^n \mathbf{1} \{ X_i = 1 \}$. The test statistic $$ Z_n = \frac{\hat{p}_n - p}{\sqrt{ \frac{\hat{p}_n (1-\hat{p}_n)}{n}}} $$ is not distributed as $Z_n \sim \mathcal{N}(0,1)$ for $n \geq 30$. The formally correct statement is that distribution of the test statistic $Z_n$ approaches a normal distribution as $n$ grows. The formal statement is that $Z_n$ approaches $Z \sim \mathcal{N}(0,1)$ in distribution.

When we construct confidence intervals, we often make the "asymptotic approximation" by pretending that $Z_n \sim \mathcal{N}(0,1)$. This is how we get $(1 - \alpha)$-confidence intervals of the form $\hat{p}_n \pm \Phi^{-1}(1 - \alpha)\sqrt{ \frac{\hat{p}_n (1-\hat{p}_n)}{n}}$, where $\Phi^{-1}: [0,1] \rightarrow \mathbb{R}$ is the quantile function of the normal distribution. We can only prove the validity of these confidence intervals asymptotically as $n \rightarrow \infty$. But, we often smudge things by saying that a finite sample size $n$ (say, 30) is close enough in practice.

---

Okay, to answer your question: your professor gave you $\sqrt{ \frac{\hat{p}_n (1-\hat{p}_n)}{n}}$ as an estimate for the variance. This is something that you can obtain from your data - it is a quantity that is known to you. On the other hand, using $\sqrt{ \frac{p (1-p)}{n}}$ would be great, but it's impractical because you do not know the true parameter $p$ from your data. So, the relevant test statistic to analyze is $\sqrt{ \frac{\hat{p}_n (1-\hat{p}_n)}{n}}$.

Both of these quantities would yield a test statistic $Z_n$ is asymptotically normal. But, again, only on of them is computable from your data.

Answer 2

Confidence intervals. Suppose you are making a 95% CI for unknown success probability $p,$ based on $n = 100$ Bernoulli trials. resulting in $x = 30$ successes. Then $\hat p = x/n = 30/100 = 0.3.$

Then the standard error of $\hat p$ is $SD(\hat p) = \sqrt{\frac{p(1-p)}{n}}.$ Also, $Z = \frac{p-\hat p}{SD(\hat p)}$ would be approximately standard normal. However, you can't evaluate $SD(\hat p)$ because $p$ is unknown.

For large $n$ you could get a good estimate of $SD(\hat p)$ of the form $\widehat{SD}(\hat p) = \sqrt{ \frac{\hat p(1-\hat p)}{n} }.$

From there you can make an approximate 95% CI for $p$ of the form $$\hat p \pm 1.96\sqrt{ \frac{\hat p(1-\hat p)}{n} }.$$

This is a Wald CI for $p$ based on the assumption that $n$ is large, and it is usually accurate enough for practical use if $n > 500.$ For very small $n,$ this style of CI. has very bad properties. For some values of $p$ the 'coverage probability' can be far from 95% (more or less than 95%, but most often lower).

A better style of CI (due to Agresti and Coull) uses $\tilde p = \frac{x+2}{n+4}$ and the formula $$\tilde p \pm 1.96\sqrt{ \frac{\tilde p(1-\tilde p)}{n+4} }.$$

For $n = 100, x = 30,$ the two CIs are as follows (computed using R as a calculator). Wald; $(0.21, 0.39);$ Agresti-Coull: $(0.22, 0.40).$

n = 100;  x = 30
p.hat = x/n
CI.w = p.hat + qnorm(c(.025,.975))*sqrt(p.hat*(1-p.hat)/n)
CI.w
[1] 0.2101832 0.3898168

p.est = (x+2)/(n+4)
CI.ac = p.est + qnorm(c(.025,.975))*sqrt(p.est*(1-p.est)/(n+4))
CI.ac
[1]  0.2189891 0.3963955

There are many styles of CIs for binomial proportions, some of them explained in the relevant Wikipedia article.

Test of hypotheses: Now, suppose you want to use data $n = 100, x = 30$ to test $H_0: p = 0.35$ against $H_a: p \ne 0.35.$ According to $H_0$ you have $p_0 = 0.35.$ The estimate from data is $\hat p = x/n = 0.3.$

The test statistic for an approximate normal test is $Z = \frac{p_0 - \hat p}{\sqrt{ \frac{p_0(1-p_0)}{n} }}.$ You reject $H_0$ at the 5% level of significance, if $|Z| \ge 1.96.$ Because $Z = 1.048,$ the null hypothesis is not rejected.

z = (.35 - .30)/sqrt(.35*(1-.35)/100);  z
[1] 1.048285

Exact procedures in R. In R statistical software you can test the hypothesis above and get a 95% CI, without using any normal approximations.

binom.test(30, 100, p=.35)

         Exact binomial test

data:  30 and 100
number of successes = 30, number of trials = 100,
 p-value = 0.3454
alternative hypothesis: 
 true probability of success is not equal to 0.35
95 percent confidence interval:
  0.2124064 0.3998147
sample estimates:
 probability of success 
                    0.3 

The null hypothesis is not rejected because the P-value $0.3454 > 0.05 = 5\%,$ and a 95% CI for $p$ is $(0.212, 0.400).$