Stuck at the beginning of the proof of uniform convergence in probability.

Question

If $X_1, \ldots, X_n$ are i.i.d. $p(x; \theta_0) \in \{p(x; \theta) : \theta\in \Theta\}$, $\Theta$ is compact, $\log p(x; \theta)$ is continuous in $\theta$ for all $\theta \in \Theta$ and all $x \in X$, and if there exists a function $d(x)$ such that $|\log p(x; \theta)| \le d(x)$ for all $\theta \in \Theta$ and $x \in X$.

and we also know that the uniform continuity also implies that

$\Delta(x, \delta) = \underset{{\{(\theta_1,\theta_2):||\theta_1−\theta_2||<\delta\}}}{\sup} \log p(x; \theta_1) − \log p(x; \theta_2)|\to 0$ as $\delta \to 0 $

My question is why do we have $\Delta(x, \delta) \le 2d(x)$ and how to prove that , by the dominated convergence theorem we have that $E_{\theta_0}[\Delta(X, \delta)] \to 0$ as $\delta \to 0$.

thank you very much in advance

Answer 1

I give here more details about the bound. Look, first, ignore the supremum and apply the triangular inequality inside. Then you simply apply the fact that the supremum over a sum of nonnegative functions is less or equal than applying the supremum to each one of the addends: $\sup_{x} (f(x) + g(x)) \leq \sup_{x} f(x) + \sup_{x} g(x) $. In particular, all together it becomes:

$$ \sup _{\left(\theta_{1}, \theta_{2}\right):|| \theta_{1}-\theta_{2} \|<\delta}\left|\log p\left(x, \theta_{1}\right)-\log p\left(x, \theta_{2}\right)\right| \leq \\ \leq \sup _{\left(\theta_{1}, \theta_{2}\right):\left\|\theta_{1}-\theta_{2}\right\|<\delta}\left|\log{p\left(x, \theta_{1}\right)}\right|+ \mid \log p\left(x, \theta_{2}\right) \mid \leq \\ \leq \sup _{\left(\theta_{1}, \theta_{2}\right):\left\|\theta_{1}-\theta_{2}\right\|<\delta} \underbrace{\left|\log{p\left(x, \theta_{1}\right)}\right|}_{\leq d(x)}+\sup _{\left(\theta_{1}, \theta_{2}\right):|| \theta_{1}-\theta_{2} \|<\delta} \underbrace{\mid \log p\left(x, \theta_{2}\right) \mid}_{\leq d(x)} \leq 2 d(x) $$

Note that in the very last passage we drop the supremums as they don't act on $d(x)$, as such function is independent of $\theta$.

(Please if this is useful, don’t forget to approve the answer and vote it up).

Answer 2

So you have your envelope $d(x)$, which is integrable (this is the standard assumption in these kind of proofs). Moreover note that the envelope is uniform, i.e. it does not depend on any $\theta$ whatsoever. Thus, using triangular inequality and splitting the supremum you get:

$$\sup_{(\theta_1 , \theta_2) : || \theta_1 - \theta_2|| < \delta} | \log{p(x,\theta_1)} - \log{p(x,\theta_2)} | \leq \sup_{(\theta_1 , \theta_2) : || \theta_1 - \theta_2|| < \delta} | \log{p(x,\theta_1)}| + \sup_{(\theta_1 , \theta_2) : || \theta_1 - \theta_2|| < \delta} | \log{p(x,\theta_2)} | \leq 2 d(x)$$

As the envelope is uniform over $\theta$.

Then you now have an integrable envelope over $ \Delta(x,\delta) $. Thus you can apply dominated convergence thm:

$$\lim_{\delta \rightarrow 0} \mathbb{E} [\Delta(x,\delta)] = \mathbb{E} [\lim_{\delta\rightarrow 0} \Delta(x,\delta)] =0 $$

Because $\lim_{\delta\rightarrow 0} \Delta(x,\delta) =0 $ almost surely.

(Please if this is useful, don’t forget to approve the answer and vote it up),