What type of drawn sample is more informative about $\theta$?

Question

We have two sampling methods:
1. $X_1,\ldots,X_n$ from a Bernoulli$(\theta)$ distribution; 2. $Y_1,\ldots,Y_n$ from a Geom($\theta$) distribution.

Which is more informative about $\theta$ and also what would you choose if the sample size $n$ is large enough?

I'mnot sure what to say. The MLE in both cases is $\bar{X}$.
The UMVUE in both cases is also $\bar{Y}$.
Now I'm not sure what to look at.

The Variance of $\bar{X}$ is $\frac{\theta(1-\theta)}{n}$
and the variance of $\bar{Y}$ is $\frac{1-\theta}{n\theta^2}$.

The Fisher information of $(X_i)_{i=1}^n$ is $\frac{n}{\theta(1-\theta)}$
and the fisher information of $(Y_i)_{i=1}^n$ is $\frac{n}{\theta^2(\theta-1)}$.

What I've observed is that the MLE of the bernoulli sample achieves the cramer rao lower bound for the variance, but this is not true for the geometric case. Also the variance for the Bernoulli distribution is always less than the Geometric's.
Not sure what else to say.

Answer 1

Observing n Bernoulli variables gives you observations of n independent events of probability $\theta$. Observing n geometric variables gives you AT LEAST n independent events of probability $\theta$, but usually a lot more since you get to observe n successes and all the failures until the nth success. However, since you always get n successes, it's not clear those are very informative, only the failures in that case, so it's not obvious you get more information. But intuitively, if you think $\theta$ is small (successes are rare, so that you observe a lot of events under the geometric), it seems you should use the geometric. Otherwise, perhaps it's not so clear.

Unfortunately, your math seems a little off for the geometric. For avoidance of doubt the following discussion will use the version of the geometric that is the number of trials up to and including the first success (i.e., supported on the positive rather than the nonnegative integers). The MLE $\hat{\theta}$ for the geometric is $1/\bar{Y}$, not $\bar{Y}$, and the Fisher information is $\frac{n}{\theta^{2}(1-\theta)}$. Since $\theta <= 1$, this is at least as great as the Fisher information for the Bernoulli, and the difference is much greater for rare events, just as one intuitively expects.

Bottom line: Prefer the geometric as long as there are not cost considerations involved. It has more information about $\theta$. However, that's not surprising since it also likely involves conducting more trials and thus more expense!

Also, you should note that the number of failures you will observe if you take the geometric approach will be negative-binomially distributed. (And the total number of trials will be the n successes plus that negative binomial.) In fact, you could view this entire problem as comparing the binonmial and negative binomial rather than the Bernoulli and the geometric.

Answer 2

Source Wikipedia:

Let $f(X; θ)$ be the pdf for $X$ given $θ$. It is also the likelihood function for $θ$. It describes the probability that we observe a given sample $X$, given a known value of $θ$.

If $f$ is sharply peaked with respect to changes in $θ$, it is easy to indicate the “correct” value of $θ$ from the data, or equivalently, that the data $X$ provides a lot of information about the parameter $θ$. If the likelihood $f$ is flat and spread-out, then it would take many, many samples like $X$ to estimate the actual “true” value of $θ$ that would be obtained using the entire population being sampled. This suggests studying some kind of variance with respect to $θ$.

This variance is exactly the Fisher information:

$ \mathcal{I}(\theta)=\operatorname{E} \left[\left. \left(\frac{\partial}{\partial\theta} \log f(X;\theta)\right)^2\right|\theta \right] $

For a given estimator $\theta_{MLE}$, the bigger $\mathcal{I}(\theta_{MLE})$ the greater is our confidence on the likelihood of our true parameter to be close to the estimator.

$\frac{\mathcal{I}(\theta_{MLE}^{\mathcal G})}{\mathcal{I}(\theta_{MLE}^{\mathcal B})} = \frac 1\theta$

Here is a roughly interpretation of this ratio. The curve of the log-likelihood is $\frac 1\theta >1$ times sharper around the maximum $\theta_{MLE}^{\mathcal G}$ than around the maximum $\theta_{MLE}^{\mathcal B}$. The sharper the better because it means that the likelihood of other values in the neighborhood are lower.