Maximum Likelihood without density?

260 Views Asked by At

The maximum likelihood-estimator of a parameter $\theta$ is defined as the maximizer $\hat \theta$ of the joint density $f$ of the data $x_1,\ldots,x_n$: $$ f(x_1,\ldots,x_n; \theta). $$

The assumption, in my understanding, is that the data actually follows the distribution belonging to that density.

Question: Is there a canonical extension of maximum likelihood to the case where the distribution of the data cannot be assumed to have a density (w.r.t. Lebesgue-measure)?

1

There are 1 best solutions below

3
On

If you're only concerned with families of distributions having no density with respect to Lebesgue measure, but possibly having densities with respect to other measures, then this is done all the time with discrete distributions. Suppose $X_1,\ldots,X_n\sim\operatorname{i.i.d. Poisson}(\lambda).$ Then the density with respect to counting measure on $\{0,1,2,\ldots\}^n$ is $$ (x_1,\ldots,x_n) \mapsto \prod_{i=1}^n \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} = \frac{\lambda^{\sum_{i=1}^n x_i} e^{-n\lambda}}{\prod_{i=1}^n x_i!} \text{ for } x = 0,1,2,3,\ldots $$ and the likelihood function is $$ \lambda \mapsto \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} = \frac{\lambda^{\sum_{i=1}^n x_i} e^{-n\lambda}}{\prod_{i=1}^n x_i!}. $$ Certainly these measures are not absolutely continuous with respect to Lebesgue measure.

More generally, suppose $X_1,\ldots,X_n\sim\operatorname{i.i.d.}$ and for every set $A$ in some sigma-algebra and every value of $\theta$ in the parameter space, the probability ${\Pr}_\theta(X_i\in A)$ is well defined.

If for every measurable set $A$ we have $\displaystyle\sum_\theta {\Pr}_\theta(X_i\in A) <\infty,$ then the set-function $\displaystyle A\mapsto \sum_\theta {\Pr}_\theta (X_i\in A),$ although not a probability measure, is a measure with respect to which every one of the probability measures ${\Pr}_\theta$ has a density function. In some cases any one of the probability measures in the parametrized family is a measure with respect to which all of the others are absolutely continuous. And I suspect in some cases a sum of two of them might be enough.