Estimators and Confidence intervals

Question

I was curious as to what the relationship between probabilistic values, estimators and confidence intervals are. I was wondering, if I have an estimator of some parameter $\lambda$, and a probability value that depended on $\lambda$ and the estimator, what would be the relationship between those two and a confidence interval for an arbitrary distribution?

Answer 1

Confidence intervals are formed by inverting probability statements pertaining to pivotal quantities. In the present context, a pivotal quantity is a function $f(\mathbf{X}, \lambda)$ with a probability distribution that is fixed (i.e., does not depend on $\lambda$). Pivotal quantities are often formed by taking an estimator of $\lambda$ and then standardising that estimator or doing some other thing to it that yields a quantity with a fixed distribution. This means that there is often (but not always) a relationship between a pivotal quantity used to form the confidence interval for $\lambda$, and an estimator of $\lambda$.

For an arbitrary distribution it is not possible to determine if this is the case, but in many particular problems it holds. Here is an example of a case where there is a connection between these things.

---

An example: A simple example of the connection between an estimator and a confidence interval is for IID normal data. If $X_1,...,X_n \sim \text{N}(\mu, \sigma)$ and we want to estimate $\mu$ then we can form the pivotal quantity:

$$\frac{\bar{X}_n - \mu}{S_n^2 / \sqrt{n}} \sim \text{Student-T}(df = n-1).$$

This pivotal quantity makes use of the point estimator $\bar{X}_n$ and "studentizes" this estimator by subtracting its mean and dividing by its estimated variance. We can use this pivotal quantity to obtain the probability statement:

$$1-\alpha = \mathbb{P} \Bigg( - t_{n-1,\alpha/2} \leqslant \frac{\bar{X}_n - \mu}{S_n^2 / \sqrt{n}} \leqslant t_{n-1,\alpha/2} \Bigg).$$

Inverting this probability statement gives:

$$1-\alpha = \mathbb{P} \Bigg( \bar{X}_n - \frac{t_{n-1,\alpha/2}}{\sqrt{n}} \cdot S_n^2 \leqslant \mu \leqslant \bar{X}_n + \frac{t_{n-1,\alpha/2}}{\sqrt{n}} \cdot S_n^2 \Bigg),$$

which gives us the standard confidence interval formula:

$$\text{CI}(1-\alpha) = \Bigg[ \bar{x}_n \pm \frac{t_{n-1,\alpha/2}}{\sqrt{n}} \cdot s_n^2 \Bigg].$$