Maximum Likelihood Estimate with Multiple Parameters

Question

I am not very familiar with multivariable calculus, but something tells me that I don't need to be in order to solve this problem; take a look:

Suppose that $X_1,...,X_m$ and $Y_1,...,Y_n$ are independent exponential random variables with $X_i\sim EXP(\lambda)$ and $Y_j\sim EXP(\theta \lambda)$.

Find the $MLE$ of $\lambda$ and $\theta$.

Finding the MLE of $\lambda$ is simple; by ignoring the $Y_j$ altogether and just looking at the $X_i$, it turns out to be $\sum x_i/m$. However, for $\theta$, I am no longer sure since the distribution of $Y_j$ is also dependent on $\lambda$. I don't know if I need to go as far as finding the gradient or if I can somehow use my previous result, but either way, I honestly don't know how to do it.

Any advice would be appreciated.

Answer 1

The log likelihood is given by $(m+n)log(\lambda) + n log(\theta)-\lambda \sum x_i -\theta \lambda \sum y_i$

The MLE for $\lambda$ including both $X$ and $Y$ turns out to be the same as just using $X$. That wasn't obvious to me.

For $\theta$ you get $n/\theta = \lambda \sum y_i$ for which you just substitute for the MLE of $\lambda$.

Answer 2

Deriving the MLE: From your specification of the problem, your log-likelihood function is:

$$\begin{equation} \begin{aligned} \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}(\theta, \lambda) &= \sum_{i=1}^m \ln p (x_i | \lambda) + \sum_{i=1}^n \ln p (y_i | \theta, \lambda) \\[8pt] &= \sum_{i=1}^m (\ln \lambda - \lambda x_i) + \sum_{i=1}^n (\ln \theta + \ln \lambda - \theta \lambda y_i) \\[8pt] &= m ( \ln \lambda - \lambda \bar{x} ) + n ( \ln \theta + \ln \lambda - \theta \lambda \bar{y}). \end{aligned} \end{equation}$$

This gives the score functions:

$$\begin{equation} \begin{aligned} \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \theta}(\theta, \lambda) &= n \Big( \frac{1}{\theta} - \lambda \bar{y} \Big), \\[8pt] \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \lambda}(\theta, \lambda) &= m \Big( \frac{1}{\lambda} - \bar{x} \Big) + n \Big( \frac{1}{\lambda} - \theta \bar{y} \Big). \end{aligned} \end{equation}$$

Setting both partial derivatives to zero and solving the resulting score equations yields the MLEs:

$$\hat{\theta} = \frac{\bar{x}}{\bar{y}} \quad \quad \quad \hat{\lambda} = \frac{1}{\bar{x}}.$$

(Note that in the case where $\bar{y} = 0$ the first of the score equations is strictly positive and so the MLE for $\theta$ does not exist.) As user121049 correctly points out, the MLE for $\lambda$ is the same as if you only used the $x_i$ values.

Answer 3

Both answers are good but in practice you'd also want to obtain some measure of precision for the estimates. One approach is to use a bootstrap. Below is another common approach.

Take the 2nd derivatives of the log of the likelihood function:

$$\frac{\partial ^2\text{logL}}{\partial \{\lambda ,\theta \}^2}=\left( \begin{array}{cc} \frac{\lambda ^2}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} & -\frac{\theta ^2 \lambda ^2 \bar{y}}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} \\ -\frac{\theta ^2 \lambda ^2 \bar{y}}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} & \frac{\theta ^2 (m+n)}{n \left(\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n\right)} \\ \end{array} \right)$$

Take minus the inverse of that resulting matrix and then substitute in the maximum likelihood estimators. One obtains

$$\hat{\Sigma}=\left( \begin{array}{cc} \frac{1}{m \bar{x}^2} & -\frac{1}{m \bar{y}} \\ -\frac{1}{m \bar{y}} & \frac{\bar{x}^2 (m+n)}{m n \bar{y}^2} \\ \end{array} \right)$$

An estimate of the variance of $\hat{\lambda}$ is $1/(m \bar{x}^2)$ and an estimate of the variance of $\hat{\theta}$ is $\frac{\bar{x}^2 (m+n)}{m n \bar{y}^2}$. An estimate of the covariance is $-\frac{1}{m \bar{y}}$.