Context: The Poisson process
Take $m$ independent and identical Poisson processes with rate parameter, $\lambda$ running in parallel (we can assume they start at the same time, but shouldn't matter given they're memory-less).
Then, take a time $u_s$. Observe all $m$ processes until time $u_e = u_s+u$.
Using the data just within this interval, what is the best estimate of $\lambda$? I show below (appendix) that this estimate is (if the observation interval length for process $i$ is $u_i$, here if so happens that $u_i=u \;\; \forall \;i$ and the number of events observed falling into interval $i$ is $n_i$).
$$\hat{\lambda} = \frac{\sum\limits_{i=1}^m n_i}{\sum\limits_{i=1}^m u_i} = \frac{\sum\limits_{i=1}^m n_i}{m u} \tag{1}$$
Using the exponential inter-arrivals instead
Now, let's say I don't know that that distribution of the counts of events lying within the intervals is Poisson distributed. All I know is that the distribution of the inter-arrival times is exponentially distributed with rate $\lambda$. I want to now use this probability distribution to get the same $\lambda$ estimate in equation (1).
Based on the Wikipedia article on censoring, the estimate of $\hat{\lambda}$ is:
$$\hat{\lambda} = \frac{k}{\sum t_j}$$
Here, $k$ is the total number of uncensored observations and $\sum_j t_j$ is the total duration of censored as well as uncensored observations.
The question: which intervals should be censored?
Now, the question arises - which intervals should we consider censored and which ones should we consider un-censored? In the picture below, three events happened inside our observation interval from one of our Poisson processes. There is the interval $x_s$, from the start-boundary of the observation interval to the very first event (shown in blue). Then there are the intervals $t_1, t_2 \dots t_{n-1}$ between the events (shown in orange). And finally, there is the interval from the last event to the end of our observation interval, $x_e$ shown in blue. There is no doubt that $t_1, t_2, \dots$ should be un-censored observations. The question is what to do with $x_s$ and $x_e$.
- If we consider $x_s$ to be uncensored (the justification is that due to the memory-less property of the exponential distribution, the distribution from some random time to the next event is the same as that between two events) and $x_e$ to be censored, we end up with the same estimator as equation (1).
- If we consider $x_s$ to be censored and ignore $x_e$, we end up with another un-biased estimator which happens to have a higher variance than the one in equation (1). This is demonstrated via simulation here.
The question: is it just a fluke that these conflicting approaches both produce unbiased estimators? What is the "right" way to use the exponential inter-arrivals to come up with an estimator? Or are both approaches "right"?
Appendix: derivation of Poisson MLE
Let the $i$th process be observed for an interval $u_i$ and the number of events from it falling into the interval be $n_i$. The likelihood function (based on Poisson PMF) becomes:
$$L(\lambda) = \prod\limits_{i=1}^m \frac{e^{-\lambda u_i} (\lambda u_i)^{n_i}}{n_i!}$$
Taking logarithm, we get the log likelihood:
$$ll(\lambda) = \sum\limits_{i=1}^m -\lambda u_i +n_i \log(\lambda u_i) - \log(n_i!)$$
Taking derivative with respect to $\lambda$,
$$\frac{\partial ll(\lambda)}{\partial \lambda} = -\sum u_i + \sum \frac{n_i}{\lambda u_i} u_i$$
Setting to zero and solving for $\lambda$,
$$\lambda = \frac{\sum n_i}{\sum u_i}$$
Note: cross-posted from here: https://stats.stackexchange.com/questions/534124/mle-for-lambda-on-the-poisson-process-using-exponential-inter-arrivals