Uniform consistency of a sum of functions with random weights?

Question

Suppose that for $k = 1,2,...$ we have $f_k(\theta)$ a known function of the $p \times 1$ vector $\theta$, and $w_k$ a random variable with $E[w_k] = 1$.

Assume that for $\theta \in \Theta$ where $\Theta$ is a compact set, we have

$$n^{-1}\sum^n_{k=1} w_k f_k(\theta) - n^{-1}\sum^n_{k=1}f_k(\theta) \xrightarrow{p}\ 0 $$

Question: Under which conditions do we also have that

$$ \sup_{\theta \in \Theta} \left| n^{-1}\sum^n_{k=1} w_k f_k(\theta) - n^{-1}\sum^n_{k=1}f_k(\theta) \right| \xrightarrow{p}\ 0 $$

that is, that the convergence in probability is uniform in $\theta$ over the compact $\Theta$?

The assumption of uniform convergence comes up often, especially for the theory of M-estimators, but I don't know how to prove it.

Answer 1

More generally, consider function $G_n(\theta)$. Assume the following:

(A1) $\Theta$ is compact

(A2) $G_n(\theta) \xrightarrow{p}\ 0, \forall \theta \in \Theta $

(A3) $G_n(\theta)$ is stochastically equicontinuous

Theorem. Under conditions A1-A3, we have

$$ \sup_{\theta \in \Theta} \left| G_n(\theta) \right| \xrightarrow{p}\ 0 $$

Proof. From the stochastic equicontinuity assumption, for any set $\epsilon > 0$, there exists a $\delta > 0$ such that

$$\limsup_{N \rightarrow \infty} P\left( \sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} | G_n(\theta) - G_n(\theta') | > \epsilon\right) = 0$$

From $A1$, a finite set $\{\theta_k\}$ exists such that the balls $B(\theta_k, \delta$) cover $\Theta$.

Now, we have

$$ \begin{align}P( \sup_{\theta \in \Theta} | G_n(\theta) | > 2\epsilon) &= P( \max_k \sup_{\theta \in B(\theta_k, \delta)} |G_n(\theta) - G_n(\theta_k) + G_n(\theta_k)| > 2\epsilon) \\ &\le P( \max_k \sup_{\theta \in B(\theta_k, \delta)} |G_n(\theta) - G_n(\theta_k)| + |G_n(\theta_k)| > 2\epsilon) \\ &\le P( \max_k \sup_{\theta \in B(\theta_k, \delta)} |G_n(\theta) - G_n(\theta_k)| > \epsilon) + P( \max_k |G_n(\theta_k)| > \epsilon) \\ &\le P( \sup_{\theta \in \Theta}\sup_{\theta' \in B(\theta,\delta)} | G_n(\theta) - G_n(\theta') | > \epsilon) + P( \max_k |G_n(\theta_k)| > \epsilon) \end{align}$$

The first term on the last line converges to zero by stochastic equicontinuity. Because of $A2$ and the fact that the set $k$ is finite, the second term also converges to zero. Because $\epsilon$ is arbitrary, the theorem is proven.

In the original problem, we may take

$$G_n(\theta) = n^{-1}\sum^n_{k=1} (w_k - 1) f_k(\theta)$$

Conditions $A1$ and $A2$ are already satisfied by assumptions. However, we would have to prove $A3$. In this case, we need that for any $\epsilon > 0$ we have a $\delta >0 $ such that

$$\limsup_{N \rightarrow \infty} P\left( \sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \left| n^{-1}\sum^n_{k=1} (w_k - 1) [f_k(\theta) - f_k(\theta')] \right| > \epsilon\right) < \epsilon$$

If we suppose

(A4) $f_k(\theta)$ are pointwise equicontinuous in $\theta \in \Theta$

(A5) $ n^{-1}\sum^n_{k=1} |w_k| = Op(1)$

Theorem. Under $A1$, $A4$ and $A5$, $G_n(\theta)$ is stochastic equicontinuous

Proof. We can write,

$$\begin{align} \sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \left| \sum^n_{k=1} \frac{w_k - 1}{n} [f_k(\theta) - f_k(\theta')] \right| &\le \sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \sum^n_{k=1} \frac{|w_k| + 1}{n} \left|f_k(\theta) - f_k(\theta') \right| \\ &\le \sum^n_{k=1} \frac{|w_k| + 1}{n} \left(\sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \max_h \left|f_h(\theta) - f_h(\theta') \right| \right) \end{align} $$

From $A5$, there exists a $M > 0$ and $N$ such that for $n > N$,

$$P\left( \sum^n_{k=1} \frac{|w_k| + 1}{n} > M \right) < \epsilon$$

From $A1$ and $A4$, the $f_k(\theta)$ are uniformly equicontinuous. Hence, there exists a $\delta > 0$ such that

$$ \sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \max_h M \left|f_h(\theta) - f_h(\theta')\right| < \epsilon$$

Hence, for $n > N$,

$$\begin{align} P\left(\sup_{\theta \in \Theta} \sup_{\theta' \in B(\theta,\delta)} \left| \sum^n_{k=1} \frac{w_k - 1}{n} [f_k(\theta) - f_k(\theta')] \right| > \epsilon \right) \le P\left( \sum^n_{k=1} \frac{|w_k| + 1}{n} > M \right) < \epsilon \end{align}$$

which proves the result.