How to interpret a p-value that's significant from Fisher's Exact test

1.7k Views Asked by At

Given a binomial distribution with p=.03, n=902, the $.025$ and $.975$ quantiles are $17$ and $38$ respectively. I interpret this to mean that a variate produced by that distribution not significant at the $5$% level when it is in that range - it's a result I'd "expect" most of the time.

Yet, doing a Fisher's exact test on the $2 \times 2$ table {{885,17},{864,38}} gets a p-value of $~.0057$, so it's significant at the 5% level.

How do I interpret this?

1

There are 1 best solutions below

2
On BEST ANSWER

It is true that the ,025 and .975 quantiles of $Binom(n = 903, p = 0.03)$ are 17 and 38, respectively. However, your conclusion about significance does not make sense because you have not stated null and alternative hypotheses. (If you were testing $H_0: p = 0.03$ against $H_A: p \ne 0.03$, then getting 17 successes in 903 trials would be borderline significant, at the 5% level, and results might go one way or another depending on whether you are doing a test that assumes normality or an exact binomial test. But all this really has nothing to do with the Fisher exact test.)

For your application of the Fisher exact test, I take the null hypothesis to be that the probabilities of successes in two binomial models are equal. In your data, it seems that there were 17 successes in a sample of (885 + 17) = 902 subjects from Population A and 38 successes in a sample of (864 + 38) = 902 subjects from Population B. Roughly speaking, your significant result says that $\hat p_A = 17/902 \approx 0.019$ and $\hat p_B = 38/902 \approx 0.042$ are sufficiently different from each other to reject $H_0: p_A = p_B$ against $H_A: p_A \ne p_B$ at the 5% significance level. However, to be precise, the probability distribution used in Fisher's Exact Test, conditional on row and column totals, is hypergeometric (not binomial).

If you need additional information on the use of the hypergeometric distribution in 'Fisher's Exact Test', please see the Wikipedia article on this test. Also, if you happen to be using R, then perhaps look at one of the explanations keyed to that software.)