Does $p$-value change with sample size?

Question

I was wondering how $p$-values change with sample size or is their any relation between the two. To my knowledge, a $p$-value denotes the probability of finding observed or more extreme results than the null hypothesis claims (typically no difference). Based on the following example, let your null hypothesis be that there exists no difference in the amount of heads and tails you flip on a fair coin, that is you flip the same exact amount and your alternative be that a difference does exist. You flip a fair coin $n = 10$ times and get $7$ heads and $3$ tails, which suggests a relatively low $p$-value. But as you flip this coin more times (say $n = 100$) and now get $45$ heads and $55$ tails, your $p$-value increases - which results in you being more likely to fail to reject the null over the alternative hypothesis.

Thus, does increasing sample size, increase $p$-values in general?

Answer 1

In general, increasing the sample size changes the p-value (not necessarily increasing it). There is a lot to discuss as to why it is so and also why ''p-hacking'' is a thing, but there is a youtube video where it is explained in simple terms:

https://www.youtube.com/watch?v=42QuXLucH3Q

Answer 2

"the probability of finding observed or more extreme results than the null hypothesis claims" is not right. The p-value is the probability that a result at least as extreme as that which was actually observed would occur given that the null hypothesis is true.

If the coin you're tossing is fair, that being what the null hypothesis says, then with very high probability the p-value will be large when the number of trials is large. If the null hypothesis is false, then the probability is $1$ that the p-value will approach $0$ as the sample size grows.

Answer 3

Whether the p-value is affected by the sample size depends on: (1) whether the test statistic for the particular hypothesis test being conducted depends on the sample size, since the p-value is a probability acquired from this test statistic; and (2) whether the distribution being used also depends on the sample size, since the p-value is a probability from this distribution. Most hypothesis tests that use the normal, student's t, chi-squared, etc. distributions have test statistics and/or distributions that depend on the sample size, so most p-values are affected by a change in n; however, this change may be unnoticeably small or surprisingly large, and is a complicated concept as per @PseudoRandom