Upper bound for Rademacher complexity

672 Views Asked by At

Given $x_1^n=\{x_1,...,x_n\}$ with $x_j\in R^p$ and $\mathcal{F}$ a class of real-valued functions defined in $R^p$, define $\mathcal{F}(x_1^n)$ as $$\mathcal{F}(x_1^n)=\{(f(x_1),...,f(x_n)):f\in\mathcal{F}\}.$$ Suppose that card$(\mathcal{F}(x_1^n))\leq (n+1)^v$ with $v\geq 1$.

For $\epsilon=(\epsilon_1,...,\epsilon_n)$ with $\{\epsilon_j\}$ i.i.d. Rademacher random variables I want to show that \begin{align*} E_{\epsilon}\sup_{f\in\mathcal{F}}\Big|\dfrac{1}{n}\sum_{j=1}^n\epsilon_jf(x_j)\Big|\leq 4\sqrt{\dfrac{v\log(n+1)}{n}}\sup_{f\in\mathcal{F}}\sqrt{\dfrac{\sum_{j=1}^nf^2(x_j)}{n}}. \end{align*} This is Lemma 4.14 from High-dimensional statistics from M.J. Wainwright.

My attempt was to use Cauchy-Schwartz: we know that $4\sqrt{v\log(n+1)}\geq 1$, so \begin{align*} \Big|\sum_{j=1}^n\epsilon_jf(x_j)\Big|&\leq 4\sqrt{v\log(n+1)}\left(\sum_{j=1}^n\epsilon_j^2\right)^{1/2}\left(\sum_{j=1}^nf^2(x_j)\right)^{1/2}\\ &=4\sqrt{v\log(n+1)}\sqrt{n}\left(\sum_{j=1}^nf^2(x_j)\right)^{1/2}.\\ \end{align*} Dividing by $n$ and taking the supremum I get \begin{align*} E_{\epsilon}\sup_{f\in\mathcal{F}}\Big|\dfrac{1}{n}\sum_{j=1}^n\epsilon_jf(x_j)\Big|\leq 4\sqrt{v\log(n+1)}\sup_{f\in\mathcal{F}}\sqrt{\dfrac{\sum_{j=1}^nf^2(x_j)}{n}}. \end{align*} So I have a $\sqrt{n}$ missing in the denominator. Is there another way to prove it?

1

There are 1 best solutions below

2
On BEST ANSWER

Let $R(\mathcal{F}):=\mathsf{E}_{\epsilon}\sup_{f\in\mathcal{F}}\left|n^{-1}\sum_{i=1}^n\epsilon_i f(x_i)\right|$. First, for any $s>0$, \begin{align} e^{snR(\mathcal{F})}&\overset{(1)}{\le}\mathsf{E}_{\epsilon}\sup_{f\in\mathcal{F}}\exp\left(s\left|\sum_{i=1}^n \epsilon_if(x_i)\right|\right) \\ &\overset{(2)}{\le} 2\sum_{f\in\mathcal{F}}\mathsf{E}_{\epsilon}\exp\left(s\sum_{i=1}^n \epsilon_if(x_i)\right) \\ &=2\sum_{f\in \mathcal{F}}\prod_{i=1}^n \mathsf{E}_{\epsilon}\exp\left(s\epsilon_if(x_i)\right) \\ &\overset{(3)}{\le} 2\sum_{f\in\mathcal{F}}\prod_{i=1}^n \exp\left(s^2(f(x_i))^2/2\right) \\ &\le 2(n+1)^v\exp\left(s^2\sup_{f\in\mathcal{F}}\sum_{i=1}^n(f(x_i))^2/2\right), \end{align} where $(1)$ holds by Jensen's inequality, and $(2)$ and $(3)$ hold because $e^{|x|}\le e^{-x}+x^x \le 2e^{x^2/2}$. Therefore, \begin{align} R(\mathcal{F})\le \frac{v\ln(2(n+1))}{sn}+\frac{s}{2n}\sup_{f\in \mathcal{F}}\sum_{i=1}^n (f(x_i))^2. \end{align} Optimizing the RHS of the last bound w.r.t. $s$, one gets $$ R(\mathcal{F})\le \frac{2}{n}\left(v\ln(n+1)\times\sup_{f\in \mathcal{F}}\sum_{i=1}^n (f(x_i))^2\right)^{1/2}. $$