Confidence Interval - Misunderstandings

Question

Possible misunderstandings of the confidence interval definition keep on bugging me from time to time, and I want to get this sorted.

More specifically, I refer to a misunderstanding I believe well explained on the mobile-Wiki page, section "Misunderstandings". (also in the regular version)

For clarifying the correct meaning of the concept, the following paragraph ensues:

A $95\%$ confidence level does not mean that for a given realized interval there is a $95\%$ probability that the population parameter lies within the interval (i.e., a 95% probability that the interval covers the population parameter).[10] According to the strict frequentist interpretation, once an interval is calculated, this interval either covers the parameter value or it does not.

So, if I repeated the experiment and calculated appropriate confidence intervals $n$ times, I would capture the parameter of interest 95% in the limit. But picking one realization, I cannot make a probability statement.

It seems analogous to me of saying:

We have a machine that, when producing $n$ balls, tends to produce 95% black balls as $n$ grows. Yet, if somebody asks you, as the machine is about to produce a ball, or after a ball is produced and kept hidden from you, what colour will it be, you cannot make a probability statement.

I find this very puzzling.

Alternatively, I find this additional source of confusion:on the same page I read the following statement:

The confidence interval can be expressed in terms of a single sample: "There is a $90\%$ probability that the calculated confidence interval from some future experiment encompasses the true value of the population parameter." Note this is a probability statement about the confidence interval, not the population parameter

I fail to appreciate the subtlety: if a statement is made on the confidence interval, saying it will encompass the true value of the population parameter, is it not necessarily also a statement on the latter made? I mean, if an interval contains a real number $a$, is this not equivalent to making a statement about the number, saying the number $a$ lies in the interval?

Thanks a lot. I checked other answers on the point but failed to understand.

Answer 1

Your confusion is not uncommon and indeed is one of the many difficulties that that scientists have with statistics. One way to think of it is the process of flipping a coin. Say you have a fair coin, then you know the probability of a heads on the first toss is $0.50$, before you've made any such toss. Now toss the coin, and say it came up heads. What is the probability of the coin coming up heads on the first toss? Well, the first toss already happened, so a heads either appeared or it did not. You can no longer make a probability statement on the first toss, because we have a realization for that trial. Because you know the coin is fair you can say things like "If I repeated the experiment many times, the proportion of heads I would see is $0.50$." We are making a probability statement on the act of flipping coins (constructing confidence intervals) in the long run. Your example of the ball producing machine is essentially right, and yes it is confusing.

Under the frequentist interpretation, populations have true, fixed parameters, and confidence intervals are random variables which produce a range of values that cover the true parameter $a$ 95% of the time. The event of interest is "contains $a$", and if the assumptions of the process are correct, then this will happen 95% of the time in the long run. Nothing more, nothing less. $a$ is not a random variable, so it doesn't make sense to put any probability statement on $a$, we can only talk about the realizations of the confidence intervals.

It may help to read about credible intervals as well, to see the differences in interpretation. What most people want is a credible interval, so they can make easily digestible statements like "with 0.95 probability, the credible interval contains the parameter of interest".

https://en.m.wikipedia.org/wiki/Credible_interval

Answer 2

Under the strict frequentist interpretation, the parameter $a$ is fixed but the confidence interval is different each time you take a sample.

The point is that you conduct the same procedure every time, but the resulting confidence interval will be still be different: same target parameter $a$, same sample size $n$, same algorithm to calculate the sample statistics (e.g. mean and variance $\bar x, s$), and the same way to construct the interval e.g. $\bar x \pm ks$ with $k$ that achieves the same desired confidence level.

For example, let's assume someone (God?) knows the true value of the parameter to be $a = 1$. A researcher might get the confidence intervals as below, after conducting multiple trials (that follow the same protocol): \begin{align} \text{trial}& & &\text{confidence interval} & &\text{contains}~a=1~\text{or not}\\ 1 & & & (0.92 ,1.02) &&\text{yes} \\ 2 & & & (0.90 ,0.99) &&\text{no}\\ 3 & & & (0.97 ,1.04) &&\text{yes}\\ 4 & & & (0.98 ,1.10) &&\text{yes}\\ \vdots &&& \vdots && \vdots \end{align} Note that even with a fixed procedure, the location (center) and width of the confidence intervals are going to be different for each trial (while the target parameter $a = 1$ is always the same).

Thus I'm going to repeat some of the points (that every teaching material says)

1. Each individual confidence interval either contains the true value, or it doesn't. It's "yes" or "no" and there's no gray area.

2. Hypothetically if the researcher can repeat the trials (with the same protocols) infinitely many times, then in the limit $95\%$ of the trials, the resulting confidence interval will cover (contain) the true value.

3. The above two bullets are true regardless of the researcher knowing the true value $a = 1$ or not. Namely, if you use your hand to cover the screen and pretend the column of "yes/no" is not there, the two statements above are still true.

4. In reality, the researcher almost never know the true value of $a$, but the researcher can still make the frequency (proportion) statement in the sense of (2), the second bullet.

5. In reality with finite resources, there is often only a single trial. The researcher can still make the (hypothetical) probability statement as in the sense of (2). Meanwhile, whether or not this single trial interval contains $a$ is never known, but we know for a fact that it either contains $a$ or it doesn't.

Answer 3

Suppose we have a fair six-sided die, which when rolled, shows a whole number between $1$ and $6$ inclusive, each with equal probability $p = 1/6$.

Moreover, suppose you are specifically interested in those outcomes where the rolled number is a $1$. You propose conducting an experiment in which you roll the die $n$ times, and count the number of times the die shows a $1$. The number of times this happens is some random variable, $X$, that is some nonnegative integer between $0$ and $n$, inclusive. It is random in the sense that each time you conduct such an experiment, you may or may not get the same result. For instance, suppose $n = 5$; you might then observe in your first experiment an outcome like $$(5, 3, 1, 2, 3), \quad \implies \quad X = 1.$$ But the second time, you might get $$(4, 3, 2, 2, 6), \quad \implies \quad X = 0.$$ But the fact remains that $$X \sim \operatorname{Binomial}(n = 5, p = 1/6), \quad \Pr[X = x] = \binom{5}{x} \left(\tfrac{1}{6}\right)^x \left(1 - \tfrac{1}{6}\right)^{5-x}, \quad x \in \{0, 1, 2, 3, 4, 5\}.$$

So far, you should not have any objection to what we have described. If you do, you are not ready to discuss inferential statistics and should review elementary probability before continuing further.

---

Now that we have this binomial model for the random number of occurrences of $1$ in $n = 5$ rolls of a fair die, we can turn our attention to the notion of estimation--that is to say, how to estimate a property of the underlying model based on realizations of a sample. This is where we transition from probability theory and descriptive statistics to inferential statistics.

Intuitively, you would postulate that as you roll the die a "large" number of times, the sample proportion of the number of $1$s observed will be approximately equal to $p$. That is not to say that you are guaranteed to always have exactly $1/6$ of your $n$ die rolls be a $1$: indeed, it is trivial to see that when $n$ is not a multiple of $6$, such an outcome is mathematically impossible. Because $X$ is a random variable, so is the sample proportion $\hat p = X/n$. It will in general vary from experiment to experiment, sample to sample. We call $\hat p$ a point estimator of the true parameter $p = 1/6$, which we regard as a fixed, unchanging property of the die.

The value of $\hat p$ can be realized by conducting an experiment, but even without doing so, we can use the underlying binomial model to make probability statements about $\hat p$ just as we did above with $X$.

The notion of a confidence interval is to simply expand upon the point estimate in a way that captures the variability of the estimator. But as with the point estimate, it too is calculated from the sample, and is therefore also a random variable: generally speaking, it is a pair of random variables $(L, U)$ for the lower and upper confidence bounds, respectively. And like the point estimate, these are liable to change from sample to sample. But the parameter remains fixed--the die has not changed in its underlying behavior.

Now, just as we can make a probability statement with the point estimate such as $$\Pr[\hat p > p],$$ we too can make a similar probability statement such as $$\Pr[L \le p \le U].$$ In fact, this expression is called the coverage probability for the confidence interval for $p$. But let us be ABSOLUTELY CLEAR: the quantities in the above statement that are random are $L$ and $U$, not $p$, just as in the point estimate probability statement, the random variable is $\hat p$, and not $p$. The difference is subtle but important, and goes back to the notion of realizing a random variable. You should not object to a statement such as $$\Pr[0.1801 > 1/6] = 1,$$ and if you know that $p = 1/6$, you could equivalently write $\Pr[0.1801 > p] = 1.$ Similarly, once we have computed $L$ and $U$ from an observed sample, say $L = 0.1357$, $U = 0.1922$, these become realized values and are no longer random, and then clearly $$\Pr[0.1357 \le p \le 0.1922] = 1$$ because $p = 1/6$ is fixed. This is the crux of your confusion. A coverage probability is a statement about the random variables that comprise a confidence interval, but once those variables are realized for an observed sample, they are no longer random, and if you know the value of the parameter, you can state with certainty whether the realized interval contains the parameter or not.

The fact that, in frequentist statistical inference, the parameters of interest are fixed but unknown, changes nothing about the fact that an expression such as $\Pr[0.1357 \le p \le 0.1922]$ can only equal $0$ or $1$, except that you cannot complete the calculation because you do not know what $p$ is. Do not confuse an unknown quantity with a random one. The former is something that does not change from experiment to experiment; the latter does. In fact, one can argue that the whole reason why statistical inference is a meaningful and interesting pursuit in a frequentist sense is because we are trying to characterize the properties of fixed but unknown parameters via estimation--for if they were not fixed, they would not be estimable; and if they were known, there is no purpose in estimating what you already know.