I have studied Hypothesis testing from thinkstats2 which does simulation to estimate p value.
For e.g. In coin toss example, we run the simulation, to generate the test statistic i.e. Difference between the number of Heads and Tails. p value is the proportion of test statistics greater than observed. So intuitively:
- if $p$ is low >> not randomness, its serious >> reject Ho
- if $p$ is high >> its randomness behind your Ha >> no reason to reject Ho
So p value is the probability of randomness behind observing the test statistic
Taking the intuition to a statistics-based approach, Instead of producing observations of test statistics (e.g. in Difference in number of heads, tails) through simulation --> In the statistical approach, we should be creating the observations/distribution through knowledge of empirical distribution.
Is my understanding correct? If this is correct then any help on how we should be creating this distribution?
Ultimately we have to answer:Is it randomness behind my observation (Ha) or something serious?