Generating smooth function from a histogram.

Question

I'm working in a census-like project, where most of our old data is stored with histograms, and this make some of the analysis we need to make difficult, so this problem arose:

Given a histogram (a sequence $(x_1,...,x_N), x_i \in \Bbb Z$), generate a function $f: \Bbb R\to \Bbb R$ with $f(i)=x_i$, $f\in C^\infty$ and $f(\pm\infty)= 0$.

Is this possible? It would be really helpful!

Note: Interpolating polynomials are no good, as they diverge at infinity.

Answer 1

The general topic of interest here is 'density estimation'. Wikipedia has a nice article (with an instructive aplet).

Of course, if you believe that the data represented in a histogram are from a parametric family, you can use information from the bin frequencies to estimate the parameters and get a 'best fitting' density curve. (For example, the $k$th noncentral moment of the data can be approximated as $\sum_{j=1}^B r_j m_j^k,$ where $m_j$ are the midpoints of the $B$ bins of the histogram and $r_j$ are the corresponding relative frequencies, $\sum_{j=1}^B r_j = 1.$)

Otherwise, you want a nonparametric approach, and the most commonly used is called kernel density estimation (KDE). I have always found Silverman's publications on KDE to be useful and clearly written.

KDE works best if you have the original data. But you might try jittering the observations in each bin to emulate the original data in such a way that KDE can give useful results. Of course, this can never be quite as good as having the original data; information is lost sorting data into histogram bins.

If you had said a little more about what you have tried with polynomials and what 'analysis [you] need', then I (or others) might be able to give advice better targeted to your project.

---

KDE is implemented in R statistical software. Below I simulated a random sample of size $n = 1000$ from $Norm(100, 15).$ The dotted blue curve is for $Norm(\bar X, S),$ where $\bar X$ and $S$ are the sample mean and standard deviation, respectively. The solid green curve is from the density estimator implemented in R (default settings). The 'rug' at the bottom shows locations of individual observations (with a lot of overplotting because of the relatively large sample size).

With 1000 observations, the two curves are usually close together, but not all results are as good as in the figure.

In case it is of use to you, here is the R code, one run of which produced the figure.

x = rnorm(1000, 100, 15); a = mean(x);  s = sd(x)
hist(x, prob=T, br=20, col="wheat", ylim=c(0,.03), main="Sample of 1000 from NORM(100, 15)")
curve(dnorm(x, a, s), lty="dotted", lwd=2, col="blue", add=T)
lines(density(x), type="l", lwd=2, col="darkgreen")
rug(x)