Upper bound on the cover number in statistical learning theory.

39 Views Asked by At

I am looking at cover numbers for statistical learning theory. Specifically, I refer to the definitions in this tutorial (Section 5.1, Definition 4).

Sticking to their definition, I read from another source that if $\mathcal{G}$ lies in a $d$-dimensional subspace of $\mathbb{R}^m$ and $\max_{g\in\mathcal{G}}\|g\|=M$, then setting $\epsilon=1/\sqrt{n}$ gives us \begin{align*} N(\mathcal{G},\epsilon,n)&\leq(2M\sqrt{dn})^d. \end{align*}

Why is this the case? I have trouble drawing the connection. Thanks.

1

There are 1 best solutions below

1
On BEST ANSWER

Your other source probably uses the Euclidean metric somewhere not the Hamming metric to get $\sqrt{d}$.

Each coordinates $x_i$ is bounded by $M$, so you can cover $[-M,M]$ with $\lceil M/(\epsilon/\sqrt{d})\rceil\leq 2M/(\epsilon/\sqrt{d})$ intervals(*) of length $2\epsilon/\sqrt{d}$ (i.e., ball in $\mathbb{R}$ of radius $\epsilon/\sqrt{d}$). So you look at the $\lceil M/(\epsilon/\sqrt{d})\rceil^d$ points in $\mathbb{R}^d$ where each coordinate is one of the centre of these intervals, and each cube with these intervals as side will be contained in a Euclidean ball of radius $\leq\sqrt{(\frac\epsilon{\sqrt{d}})^2+\dots+(\frac\epsilon{\sqrt{d}})^2}=\epsilon$.

(*) closed intervals and closed balls here. If you want the open version you need to insert some $+1$s to make sure internal boundaries are covered.