MLE : Effect of incorrect variance on the mean for a normal distribution

87 Views Asked by At

Consider we have univariate samples, $x_k$, belonging to a category $\omega$ drawn from a dataset D according to an assumed distribution $p(x|\omega)$ $\sim N(\mu, 1)$. However, let the true distribution be $p(x|\omega) \sim N(1, 10^6)$.

How is the maximum likelihood estimate of $\mu$ going to be affected by the fact that the value of variance that is assumed known is incorrect?

The MLE estimate for the mean of a normal distribution, $\mu = \frac{1}{N}\sum_1^Nx_k$ which is not affected by the fact that the assumed variance is incorrect.

1

There are 1 best solutions below

0
On

You answered yourself! The maximum likelihood is independent of the variance of the noise.

Since you have already answered the question, I'm gonna show you that even if you estimate the noise variance together with the mean, the estimator of the mean is going to be the same.

Consider the likelihood $L(\mu, \sigma^2) = \sum_{k=1}^N \log p(x_k)$, we have that $$L(\mu,\sigma) = \frac{1}{2\sigma^2} \sum_{k=1}^N (x_k - \mu)^2 + N \log \sigma^2 + const.$$

Maximize with respect to $\sigma^2$, you have that the MLE of $\sigma^2$ is $$\hat \sigma^2(\mu) = \frac{1}{N}\sum_{k=1}^N (x_k - \mu)^2$$ that is, the (biased) sample variance. Plug this back into $L(\mu,\sigma^2)$ and you get $$L_2(\mu) = L(\mu,\hat \sigma^2(\mu)) = \log[ \hat \sigma^2(\mu)] + const = \log\left( \frac{1}{N}\sum_{k=1}^N (x_k - \mu)^2\right) + const;$$ Since the logarithm is monotone increasing, maximizing the log is equivalent to maximizing the argument, so we have that the MLE of the mean is again: $$\hat\mu = \frac{1}{N} \sum_{k=1}^N x_k.$$