I was expecting a t-test on two sets of data generated by the same distribution to pretty much have a p-value of 1. I was surprised to see it could be much, much lower. Can somebody please explain why?
My Python code to demonstrate looks like this:
import scipy.stats as stats
import numpy as np
Nsamp = 10 ** 6
sigma_x = 2
mean = 100
ps = []
for x in range(10):
x = np.random.normal(mean, sigma_x, size=Nsamp)
y = np.random.normal(mean, sigma_x, size=Nsamp)
(t_value, p_value) = stats.ttest_ind(x, y, equal_var=True)
ps.append(p_value)
p_mean = sum(ps) / len(ps)
print("p-values: Average {0:.3f} lowest {1:.3f}".format(p_mean, min(ps)))
And it's not unusual for me to see something like:
p-values: Average 0.553 lowest 0.088
EDIT: It's also worth mentioning that a chi-square test on the same data consistently returns a p-value of 1.0.
What you're seeing is what you should expect to see.
When a point-null hypothesis is true and the statistic has a continuous distribution (and the assumptions of the procedure all hold), then p-values should have a standard uniform distribution. i.e. they are exactly as likely to be near 0 as they are to be near 1. That this is true follows immediately from the definition of a p-value.
For example here's the p-values from ten thousand (equal-variance-) two-sample t-tests where the data came from standard normal distributions ($n_1=n_2=10$):
[As I said above, this uniformity is a direct consequence of the definition of the p-value.]
Under the alternative (if your test has power against the alternative), the p-values "crowd" down toward zero more but you can still observe occasional large p-values:
Here the difference in means was 1/5 of a standard deviation, but otherwise the conditions were as for the first histogram.