Derive the Expected Value For Maximum Likelyhood Estimators of Mean and Standard Deviation

Question

I'm working through Bishop's Pattern Recognition and Machine Learning. On Page 27 he derives the expressions for the maximum likelihood estimators of the mean and standard deviation for a Gaussian distribution:

$\mu_{ML} = \frac{1}{N}\sum_{n=1}^Nx_n$
$\sigma^2_{ML} = \frac{1}{N}\sum_{n=1}^N(x_n-\mu_{ML})^2$

He then goes on calculate expectation values of those same quantities

$\mathbb{E}\left[\mu_{ML}\right]=\mu$
$\mathbb{E}\left[\sigma^2_{ML}\right] = \left(\frac{N-1}{N}\right)\sigma$

How do you derive the expected values for these quantities?

Answer 1

1. $$ \mathbb{E}[\hat{\mu}] = \mathbb{E}\frac{1}{n}\sum X_i =\frac{1}{n}\sum \mathbb{E}X_i=\frac{n}{n}\mu = \mu. $$

2. Recall that if $X_1,..,X_n \sim \mathcal{N}(\mu, \sigma^2)$, then $$ \frac{\sum (X_i - \bar{X}_n)^2}{\sigma^2} \sim \chi^2_{(n-1)}, $$ hence, $$ \mathbb{E}\left[\frac{\sum (X_i - \hat{\mu})^2}{n}\right] = \mathbb{E}\left[\frac{\sigma^2}{n}\frac{\sum (X_i - \hat{\mu})^2}{\sigma^2}\right] = \frac{\sigma^2(n-1)}{n}. $$

Answer 2

Here's a more mechanical approach for computing $E[\sigma_{ML}^2]$ that you'd see in a standard stats textbook:

$$E[\sigma_{ML}^2] = \frac{1}{n}E[\sum_{i=1}^n(X_i - \bar X)^2]=\frac{1}{n}(E[\sum_{i=1}^nX_i^2 - 2X_i \bar X + \bar X^2])$$

$$E[\sigma_{ML}^2]=\frac{1}{n}(E[\sum_{i=1}^nX_i^2] - 2E[\bar X\sum_{i=1}^nX_i] + nE[\bar X^2]) = \frac{1}{n}(E[\sum_{i=1}^nX_i^2] - 2nE[\bar X^2] + nE[\bar X^2])$$

Where we use: $\sum_{i=1}^nX_i = n\bar X$. Now we have:

$$E[\sigma_{ML}^2]=\frac{1}{n}(n(\sigma^2 + \mu^2) - nE[\bar X^2])$$

Because $E[X_i^2] = \sigma^2 + \mu^2 $.

Crucial step:

$E[\bar X^2] = Var(\bar X) + E[\bar X]^2 = Var(\frac{1}{n}\sum_{i=1}^nX_i) + \mu^2 = \frac{1}{n^2}(n\sigma^2) + \mu^2 = \frac{\sigma^2}{n} + \mu^2$

Where we used the fact that the $X_is$ are independent. Back to $\sigma_{ML}^2$:

$$E[\sigma_{ML}^2] = \sigma^2 + \mu^2 - n\frac{\sigma^2}{n} - n\mu^2 = \frac{(n-1)\sigma^2}{n}$$

Note that if you don't want to use the trick $E[\bar X^2] = Var(\bar X) + E[\bar X]^2$ and be more mechanical in your approach:

$$E[\bar X^2] = \frac{1}{n^2}E[(\sum_{i=1}^nX_i)^2] = \frac{1}{n^2}E[\sum_{i=1}^nX_i^2 + \sum_{i \not = j}^nX_iX_j] = \frac{1}{n^2}(E[\sum_{i=1}^nX_i^2] + \sum_{i \not = j}^nE[X_i]E[X_j])$$

By independence of $X_i$ and $X_j$.

$$E[\bar X^2] = \frac{1}{n^2}(n(\sigma^2 + \mu^2) + n(n-1)(\sigma^2 + \mu^2)) = \frac{1}{n^2}(n\sigma^2 + n^2\mu^2) = \frac{\sigma^2}{n} + \mu^2$$