hypothesis testing and CI

Question

when can't we test the hypothesis testing based on confidence interval and when can we?
I just know that one example is since we have used estimator of p to estimate the normal distribution but we use p null in hypothesis testing, they are not equivalence. so, we cannot test the hypothesis testing based on confidence interval.
so, what's the general method to determine this question?

Answer 1

Suppose you wanted to test the hypothesis of whether two populations have the same mean, and you take samples from each.

You might develop a confidence interval for the difference between the means, and see if $0$ is in that confidence interval, and that might be satisfactory.

Or you could develop confidence intervals for each of the means and see whether the two confidence intervals overlap. That would be unlikely to give you the sort of hypothesis test you are looking for, especially in terms of a specified significance.

Answer 2

Some frequently used confidence intervals are essentially derived by 'inverting the test'. That is, the a 97% CI can be viewed as a set of non- rejectable null hypothetical values in a two-sided test at the 5% level.

For t procedures the CI is compatible with the test. For example, consider the following sample of $n = 20$ normal observations, We do a one-sample t test of $H_0: \mu = 105$ against $H_a: \mu \ne 105.$

set.seed(504)
x = rnorm(20, 100, 15)

summary(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  89.84   94.95  107.54  107.87  116.39  147.98 

The sample mean $\bar X = 107.87$ differs from $\mu_0 = 105,$ but not by enough to be considered significantly different at the 5% level because the P-value of the test is $0.503 > 0.05 = 5\%.$

t.test(x, mu = 105)

        One Sample t-test

data:  x
t = 0.85543, df = 19, p-value = 0.403
alternative hypothesis: true mean is not equal to 105
95 percent confidence interval:
 100.8409 114.9087
sample estimates:
mean of x 
 107.8748 

Notice that the 95% t confidence interval $(100.84, 114.91)$ contains $\mu_0 = 105,$ which is another signal that $H_0$ is not rejected.

Commonly used binomial tests and CIs are not always compatible. However, not all CIs in common use are precisely inverted tests. One example is for inference about a binomial success probability $p.$

Suppose we have $x = 40$ Successes in $n = 100$ independent trials and we make the Wald 95% CI for $p,$ which is of the form $\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{100}},$ where $\hat p = x/n = 0.4.$ This CI computes to $(0.304, 0.496).$

Another commonly used 95% CI for $p$ is the Agresti interval, which is of the form $\tilde p \pm 1.96\sqrt{\frac{\tilde p(1-\tilde p)}{104}},$ where $\tilde p = \frac{x+2}{n+4} = 0.4038462.$ The Agresti interval computes to $(0.309, 0.498).$

p.hat=40/100; p.hat + qnorm(c(.025,.975))*sqrt(p.hat*(1-p.hat)/100)
[1] 0.3039818 0.4960182

Notice that neither of these intervals contain $p_0 = 0.5.$

However, the procedure prop.test in R does not quite reject $H_0; p = 0.5$ against $H_a: p \ne 0.5,$ at the 5% level. The P-value is $0.057 > 0.05 = 5\%.$ So there is a disagreement between the test and the Wald interval (which does not exactly invert this test).

prop.test(40, 100, p = .5)

    1-sample proportions test with continuity correction

data:  40 out of 100, null probability 0.5
X-squared = 3.61, df = 1, p-value = 0.05743
alternative hypothesis: true p is not equal to 0.5

There are at least half a dozen styles of 95% CIs for binomial $p$ in common use. The R output of the procedure prop.test continues on to show the 95% CI $(0.305, 0.504),$ which does very nearly arise from inverting the test.

Note: The procedure binom.test in R is an exact test (not using a normal or chi-squared approximation), and it shows an exact 95% CI derived from binomial CDFs. So the CI is is an exact inversion of the test. However, many elementary statistics texts avoid these exact procedures because they are a little messy to compute.

binom.test(40, 100, p=.5)

        Exact binomial test

data:  40 and 100
number of successes = 40, number of trials = 100, p-value = 0.05689
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.3032948 0.5027908

sample estimates: probability of success 0.4