Basically, I am having a hard time understanding the difference between the $p$-value, the significance level and the size of a test.
In doing questions about hypothesis testing, I am somehow able to get the right answers when asked to derive these quantities. But at the end of the day, it seems to me that all three of them represent the probability of rejecting the null hypothesis?
Any enlightenment would be much appreciated.
A hypothesis test's significance level $\alpha$ should be set before undertaking the test, and should be the desired probability of the test rejecting the null hypothesis, given that it were true.
This has sometimes also been called the size of the test, but that usage is now less common since size is also used for other terms such as sample size and effect size; alternatively, sometimes it has been used to measure the actual probability of being in the critical rejection region (e.g. a count which is assumed to have a binomial distribution may not exactly match the significance level, and so the size could be slightly less than the significance level) and more generally, size has been used when the actual critical rejection region is chosen before the significance level
The $p$-value is the probability under the null hypothesis of seeing a result as extreme or more extreme than the observed result, with the alternative hypothesis suggesting what extreme means; the $p$-value is calculated after seeing the observation. If the $p$-value is less or equal to $\alpha$ then the test is statistically significant in terms of rejecting the null hypothesis with a significance level of $\alpha$ (not of $p$)