MLE + Cramér-Rao bound of a discrete random variable

Question

Can someone explain how to compute the maximum likelihood estimator, the Fisher information and the Cramér-Rao bound of a discrete random variable please?

I came across this exercise while reviewing for my exam but can't solve it, I had only been dealing with continuous distributions so far.

Let $(X_1,...,X_n)$ be an i.i.d. random sample of size $n$ and distribution $\mathbb{P}_\theta$.

$$\mathbb{P}_\theta(\{0\})=1/4,\hspace{20pt}\mathbb{P}_\theta(\{1\})=3\theta/4,\hspace{20pt}\mathbb{P}_\theta(\{2\})=3(1-\theta)/4$$

$\theta\in[0,1]$

Here's what I've done:

$\mathcal{L}(x_1,...,x_n;\theta)=\left\{\begin{array}{cl} \dfrac{1}{4}&\text{if } x=0\\ \dfrac{3\theta}{4}&\text{if } x=1\\ \dfrac{3(1-\theta)}{4}&\text{if } x=2\\ \end{array}\right.$

$\ln(\mathcal{L}(x_1,...,x_n;\theta))=\left\{\begin{array}{cl} \ln(1/4)&\text{if } x=0\\ \ln(3/4)+\ln(\theta)&\text{if } x=1\\ \ln(3/4)+\ln(1-\theta)&\text{if } x=2\\ \end{array}\right.$

$\frac{\partial \ln(\mathcal{L}(x_1,...,x_n;\theta))}{\partial{\theta}}=\left\{\begin{array}{cl} 0&\text{if } x=0\\ 1/\theta&\text{if } x=1\\ -1/(1-\theta)&\text{if } x=2\\ \end{array}\right.$

But then I can't figure out the MLE, the Fisher information and the Cramér-Rao bound… Thanks.

Answer 1

As I mentioned in a comment, the likelihood can be written in the following convenient way:

\begin{align} L(\theta\mid \boldsymbol x)&=\left(\prod_{i: x_i=0}\frac14\right)\left(\prod_{i:x_i=1}\frac{3\theta}4\right)\left(\prod_{i:x_i=2}\frac{3(1-\theta)}4\right)\quad,\,\theta \in [0,1] \\\\&=\left(\frac14\right)^{n_1}\left(\frac{3\theta}4\right)^{n_2}\left(\frac{3(1-\theta)}4\right)^{n_3}\,, \end{align}

where $n_1,n_2$ and $n_3$ are respectively the number of observations equal to $0,1$ and $2$ in the sample, such that $n_1+n_2+n_3=n$. These $n_i$'s are of course based on the sample observations, which can be explicitly seen by writing them as sums of indicator variables:

$$n_1(\boldsymbol x)=\sum_{i=1}^n \mathbf1(x_i=0)\,,\,n_2(\boldsymbol x)=\sum_{i=1}^n \mathbf1(x_i=1)\,,\,\text{etc.}$$

In fact, it is clear from the likelihood that $(n_1(\boldsymbol X),n_2(\boldsymbol X),n_3(\boldsymbol X))$ has a multinomial distribution.

This should help you in finding the Fisher information.

Answer 2

Or you can compute the second derivative of the log likelihood $$ \frac{\partial^2 ln L}{\partial \theta^2} = -\frac{3}{4}\left(\frac{N_2}{\theta^2} + \frac{N_3}{(1-\theta)^2}\right) $$ and take its expectation while noticing that $E(N_2) = 3 n \theta /4$ and $E(N_3) = 3n(1-\theta)/4$. You reteive the desired result since $I_n(\theta) = -E\left(\frac{\partial^2 ln L}{\partial \theta^2}\right) = n I_1(\theta)$ the Fisher information.