How to measure distance of value in Hypothesis test?

Question

If Null hypothesis is $p_1 =p_2$ and Alternative hypothesis is $p_1 <p_2$ then the p value can only show us the how strong the evidence is to support or against the null hypothesis. But what test we can use to test the difference of two values? For example p1 is 3 and p2 is 3.5 vs p1 is 3 and p2 is 80 we may get same $p$ value = 95%

Answer 1

An approximate normal test of the difference between two binomial proportions.

It is not clear to me what sample sizes and numbers of successes you have. Perhaps it is easiest to explain this in terms of a specific numerical example, in which this information is available:

Suppose you have two political parties and you want to know whether the proportion of women is the same in each. You sample 160 people from Party 1 and get 81 women, and you sample 180 from Party 2 and get 99 women. Then estimated proportions $\hat p_1 = 81/160 = 0.506$ and $\hat p_2 = 99/180 =0.55.$ You want to test $H_0: p_1 = p_2$ against $H_a: p_1 \ne p_2.$

We have $E(\hat p_1) = p_1$ and $Var(\hat p_1) = \frac{p_1(1-p_1)}{n_1}$ and similarly for $\hat p_2.$ Using the normal approximation to each binomial proportion, approximating $p_i$ by $\hat p_i$ and standardizing the difference $\hat p_1 - \hat p_2,$ we have the (approximate) test statistic $$Z = \frac{\hat p_1 - \hat p_2}{SE(\hat p_1 - \hat p_2)},$$ where $SE(\hat p_1 - \hat p_2) = \sqrt{\frac{\hat p_1(1-\hat p_1)}{n_1}+\frac{\hat p_2(1-\hat p_2)}{n_2}}$ and $Z \sim \mathsf{Norm}(0,1)$ under $H_0.$ [The sample sizes $n_1 - 160$ and $n_2 = 180$ are sufficiently large that the normal approximation is reliable, especially since $\hat p_1$ and $\hat p_2$ are near 1/2.]

In Minitab statistical software the test looks like this:

Test and CI for Two Proportions 

Sample   X    N  Sample p
1       81  160  0.506250
2       99  180  0.550000

Difference = p (1) - p (2)
Estimate for difference:  -0.04375
95% CI for difference:  (-0.149973, 0.0624731)
Test for difference = 0 (vs ≠ 0):  Z = -0.81  P-Value = 0.420

Because the test statistic has $|Z| = 0.81 < 1.96$ we cannot reject the null hypothesis that parties have the same proportions of women--at the 5% level of significance. The sample proportions do differ, but not by a statistically significant amount.

An alternative way to reach the same conclusion is to note that the P-value $0.42 > .05.$ The P-value is the total $P(Z \le -0.81) + P(Z \ge 0.81),$ where $Z \sim \mathsf{Norm}(0,1);$ that is, the probability assuming $H_0$ to be true, of observing a discrepancy in sample proportions as large as what we had in our samples or larger.

Finally, in terms of confidence intervals, we note that the 95% confidence interval includes $0,$ so that it is reasonable to say that data are consistent with no difference in proportions. The confidence interval is computed as $\hat p_1 -\hat p_2 \pm 1.96SE(\hat p_1 - \hat p_2).$