Calculate p-val of new probability vs old probability

Question

I have two distinct probabilities, the old one is p1=0.72 and the new one is p2=0.84. I have to calculate the p-value of the benefit of the new probability. How do I calculate this?

Answer 1

Suppose you have two groups 1 and 2. The respective observed success rates are $\hat p_1 = x_1/n_1$ and $\hat p_2 = x_2/n_2,$ where $x_i$ is the number of successes out of $n_i$ in group $i.$ Then you want to test $H_0: p_1 = p_2$ against $H_2: p_i \ne p_2$ at the 5% level of significance. That is, you want a test to compare these two binomial proportions. You can google 'test of binomial proportions' for various versions of such tests.

One of them is prop.test in R statistical software. If you have $x_1 = 76$ successes out of $n_1 = 200$ trials in Group 1 and $x_2 = 99$ successes out of $n_2 = 310$ trials in Group 2, then the test gives the results shown below. [I have declined the continuity correction because sample sizes, essentially in the hundreds, are sufficiently large.]

prop.test(c(76,99), c(200,310), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(76, 99) out of c(200, 310)
X-squared = 1.9837, df = 1, p-value = 0.159
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.02431843  0.14560875
sample estimates:
   prop 1    prop 2 
0.3800000 0.3193548 

The P-value $0.159 > 0.05 = 5\%$ indicates that the proportions $\hat p_1 = 0.38$ and $\hat p_2 = 0.32,$ while different, are not enough different to be called significantly different at the 5% level. With samples of this size, one might find sample proportions $\hat p_i$ this far apart about 16% of the time--even if there is no real difference between $p_1$ and $p_2.$ A 95% confidence interval for $p_1 - p_2$ is $(-0.024, 0.146),$ which includes $0.$

This test in R is very nearly the same as a chi-squared test of the contingency table below, for different proportions of successes. In the table TAB below, rows are for Successes and Failures and Columns are four Groups 1 and 2.

suc = c(76, 99);  n = c(200, 310)
fai = n - suc
TAB = rbind(suc,fai);  TAB
    [,1] [,2]
suc   76   99
fai  124  211

Then a 'chi-squared test' for a $2 \times 2$ table, in R gives the output below. If you are not familiar with such chi-squared tests and if they are not explained in your statistics text, then you can google that as well.

chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 1.9837, df = 1, p-value = 0.159

Notice that the P-value is the same as in the test of binomial proportions shown above. The former may be easier to understand and the chi-squared test may be more widely explained in textbooks and online.

Notes: (1) As @MarkoLalovic has commented: You really can't compare two binomial proportions without knowing the sample sizes $n_1$ and $n_2.$ If you had exactly the same proportions with three times as many observations in each group, then those proportions would be very highly significantly different with a P-value near $0.$

prop.test(c(226,249),c(600,930), cor=T)$p.val
[1] 9.02481e-06

(2) If there is something in the context of this problem/information that leads you to believe that there were at least 100 trials in each of the two groups, then you have a significant difference at level 6% level, for the two proportions you provide:

prop.test(c(84, 72),c(100,100), cor=T)$p.val
[1] 0.06042645

(3) if the old value $p = 0.72$ is well-established, and you are comparing new proportion $\hat p = 84/100,$ then an exact binom.test in R shows strong evidence that the new value is larger:

binom.test(84, 100, p=.72, alt="greater") 

    Exact binomial test

data:  84 and 100
number of successes = 84, number of trials = 100, 
 p-value = 0.003671
alternative hypothesis: 
 true probability of success is greater than 0.72
95 percent confidence interval:
  0.7671842 1.0000000
 sample estimates:
 probability of success 
                   0.84 

Even $\hat p = 42/50 = 0.84$ differs significantly from $0.72$ at the 5% level. [But not $\hat p = 21/25.]$

binom.test(42, 50, p=.72, alt="greater")$p.val
[1] 0.03645886

binom.test(21, 25, p=.72, alt="greater")$p.val
[1] 0.1303708