Confidence interval for sample

Question

I have a sample of size $n=19593$ of count data

  1      2     3    4    5   6   7   8   9   10  11  12  13  14  15  16  17  18  19 20 21 22 23 24 25 26 27  
18890  3032  1542  727  599 530 409 363 298 266 261 273 231 268 129 131 145 120 119 29 62 48 0  18 0  38 10 

I want to find a confidence interval for the center of the data, specifically in the form $n \mu$. Do I have enough data to estimate $\sigma$ and use the central limit theorem. The negative binomial seems to be the best fit among known distributions, but I don't think it is the right fit because of the long tail in the data, so would bootstrapping be a better approach and using the median instead of the mean to measure the center of the data.

Answer 1

Confidence intervals for mean of a highly skewed population using a large sample.

This seems to be a followup from a previous post in which you started asking about CIs for the population mean in addition to your original question (about retrieving data from confidence intervals for the frequencies of various values). Here the values $y$ and their respective frequencies $f$ are provided.

Your explicit question in the current post is about a comparison of a traditional confidence interval for the population mean $\mu$ based on the CLT and a nonparametric bootstrap. As @Bey speculates, there is good agreement.

Traditional confidence interval. First, the traditional CI. We begin by inputting your data and checking for formatting errors.

 y = c(1, 2,  3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
       16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27) 
 f = c(18890, 3032, 1542, 727, 599, 530, 409, 363, 298, 266, 261,  
       273, 231, 268, 129, 131, 145, 120, 119, 29, 62, 48, 0, 18,
       0, 38, 10) 
 n = sum(f);  x = rep(y, times=f);  n
 ## 28538  # n
 obs.mean = sum(y * f)/n;  obs.mean;  mean(x) #check
 ## 2.740872  # x.bar
 ## 2.740872  # agreement: data correctly parsed
 obs.sd = sd(x);  pm = c(-1,1);  z.star = qnorm(.975)
 obs.mean + pm*z.star*obs.sd/sqrt(n)  # trad'l CI
 ## 2.696125 2.785619

So the standard CI of the population mean $\mu,$ assuming normality is $(2.70, 2.78).$ The distribution is indeed very seriously skewed, but the sample size is huge and the traditional interval should be OK.

Nonparametric bootstrap CI. Now to compare with results of a nonparametric bootstrap. I am bootstrapping the distribution of the variability $V = \bar X - \mu,$ of the sample mean about the population mean for (very large) samples of size $n = 28,538.$

If we knew the distribution of $V,$ we could find $v_L$ and $v_U$ cutting 2.5% of the probability from the respective tails of its distribution so that $$P(v_L \le V = \bar X - \mu \le v_U) = P(\bar X - v_U \le \mu \le \bar X - v_L) = 0.95,$$ and $(\bar X - v_U, \,\bar X - v_L)$ is a 95% CI for $\mu.$

The resulting CI turns out to be $(2.69, 2.79),$ essentially identical to the result based on the CLT. The bootstrap interval is a little longer because here we are making no assumptions about the distribution of the population (except that it has a mean). So it does not matter whether the data are Poisson, negative binomial, or from some other severely skewed distribution. (The code below assumes that the code above has been run just previously. In typical bootstrap notation, $*$'s are used for resampled quantities.)

 # Bootstrap World--Estimated Distribution of V
 B = 1000;  re.x = sample(x, B*n, repl=T)
 RDTA = matrix(re.x, nrow=B)          # B x n matrix of resamples
 re.mean = rowMeans(RDTA)             # vector of B `x-bar-star's
 re.v = re.mean - obs.mean            # vector of v-star's
 v.UL = quantile(re.v, c(.975,.025))  # estimated quantiles of V

 # Real World--Bootstrap Confidence Interval
 obs.mean - v.UL
 ##    97.5%     2.5% 
 ## 2.692651 2.785346 

Notice that no estimate of the variance is used in the bootstrap procedure. The variability of $V$ is 'deduced' from repeated sampling.

Answer 2

You are right - your distribution is definitely not normal.

There seems to be a misconception in some of the answers - large $N$ does not imply the distribution is close to normal! The empirical distribution will converge to the actual distribution for a large $N$ - and not normal if the actual distribution isn't. (Only means converge to normal according to CLT.)

---

For your distribution, however, you can find confidence intervals by taking percentiles - for example the range between 2.5th percentile and 97.5th percentile is a 95% confidence interval.

The picture below illustrates this on your distribution.

Or in general, take $1-\alpha$ confidence interval by choosing $\hat F_{\alpha/2}$.