Why is the MLE a special case of the minimum contrast estimator?

Question

In my statistics lecture, we had two definitions, namely

Let $X_1,\ldots.X_n$ be iid random variables, each with density $p_{\Theta_0}(x)$. Furthermore, let $\varrho$ be a real function such that $$E_{\Theta_0}[\varrho(x,\Theta)] > E_{\Theta_0}[\varrho(x,\Theta_0)] ~~~~ \forall \Theta\neq\Theta_0. \tag{1}$$ Then, $$\hat{\Theta}=\operatorname{argmin}_{\Theta} \frac{1}{n} \sum_{i=1}^n \varrho(x_i,\Theta)$$ is the Minimum Contrast Estimator (MCE) of $\Theta$.

and

Let $X_1,\ldots.X_n$ be iid random variables, each with density $p_{\Theta_0}(x)$. Then, $$\hat{\Theta}=\operatorname{argmax}_{\Theta} \log \prod_{i=1}^n p_\Theta(x_i)$$ is the Maximum Likelihood Estimator (MLE) of $\Theta$.

My question. I do understand that setting $\varrho(x,\Theta):=-\log p_\Theta(x)$ will yield the MLE as a special case of the MCE. But why does the so definied function $\varrho$ satisfy condition (1)?

My attempt. Writing out condition (1) for this choice of $\varrho$ yields $$\int -\log( p_\Theta(x)) p_{\Theta_0}(x) dx > \int -\log( p_{\Theta_0}(x)) p_{\Theta_0}(x) dx $$ which is equivalent to $$ \int \log\left(\frac{p_{\Theta_0}(x)}{p_\Theta(x)}\right) p_{\Theta_0}(x)dx > 0$$ meaning $$E\left[ \log\left(\frac{p_{\Theta_0}(X)}{p_\Theta(X)}\right) \right] > 0.$$ I just don't know why this should be the case. Any hint is much appreciated!

Answer 1

You are almost there. What you got

$$\int \log\left(\frac{p_{\Theta_0}(x)}{p_\Theta(x)}\right) p_{\Theta_0}(x)dx = E_{{\Theta_0}}\left[ \log\left(\frac{p_{\Theta_0}(X)}{p_\Theta(X)}\right) \right] $$

is the Kullback-Leibler distance (or information divergence) (or relative entropy) between the two densities. That this is non-negative is a fundamental property, which can be proved with the Jensen inequality, or with the log-sum inequality - see eg here or here.

Using Jensen: because $-\log(\cdot)$ is convex, we have $E[-\log(g(X))] \ge - \log(E[g(X)]$

Then $$E_{{\Theta_0}}\left[ -\log\left(\frac{p_{\Theta}(X)}{p_{\Theta_0}(X)}\right) \right] \ge - \log \int p_{\Theta_0}(x) \frac{p_{\Theta}(x)}{p_{\Theta_0}(x)} dx= -\log 1 = 0$$