When conducting confidence intervals, hypothesis testing, and ANOVAs are we using the sampling distribution with multiple samples as opposed to a single sample?
Are there cases where we use just a single sample, or are we always using the sampling distribution to do the aforementioned tests?
Second question - If we are using a sampling distribution how many samples makes that distribution effective?
Thank you
First question: One observation is not enough because it give you no idea of the variance (or SD) and you need that for a test or for a CI. (If you already know the variance, then you might make a CI, but it would probably be too long to be of any use; see answer to second question below.)
Second question: This gets into more complicated territory. It's easier to discuss for CIs. For the moment consider we're sampling from a normal population with known SD $\sigma_0$ and unknown mean $\mu.$ Then a 95% CI for $\mu$ is $\bar X \pm 1.96 \sigma_0/\sqrt{n}.$ The length of this CI is $3.92 \sigma_0/\sqrt{n}.$ A hugely long CI isn't of much use. If you're trying to make a small CI, then the bigger $\sigma_0$ the bigger $n$ must be. (You could also make the CI shorter by using a lower confidence level. For example, with the same $\sigma_0$ and $n$, a 90% CI is shorter than a 95% CI.
The same issues arise if $\sigma$ is unknown. Then you'll use the t distribution to find the CI. The details are a bit more complicated, but the same trade-offs need to be made.
You don't have much control over the variance. So three factors are in tension with one another: (a) confidence level (big is good), (b) sample size (small is good), and (c) length of CI (small is good). The only way you can improve one of the three is to make one or both of the other two worse.
The similar problem for testing is a matter of computing the power of the test. If you know about that, let me know and I'll post an addendum to this Answer.