A complete orthonormal basis for every RKHS?

203 Views Asked by At

If I understand correctly, the Mercer Theorem states that if $R$ is a continuous positive-type function, then there exists a sequence of eigenfunctions $\phi_n(.)$ and corresponding nonnegative eigenvalues $\lambda_n$ associated with $R$ as an operator on $L^2$, such that $$R(s,t)=\sum_{n=1}^\infty \lambda_n\phi_n(s)\phi_n(t)$$

Let $\mathcal{H}$ be the reproducing-kernel Hilbert space (RKHS) corresponding to $R$ (cf Moore-Aroszajn Theorem). I am reading the section on RKHS in Elements of Statistical Learning and Inference by Hastie et al (2017 edition, https://hastie.su.domains/ElemStatLearn/, p.168), and after positing an above expansion for $R$, the authors state that $$\forall f\in\mathcal{H}\hspace{0.6cm} f(x)=\sum_{i=1}^{\infty}c_i\phi_i(x)$$ where $\phi_i$ are the eigenvalues associated with $R$, and $c$ are suitable scalars.

Here's my problem. The latter statement implies that $\mathcal{H}$ has a countable orthonormal basis, which means that $\mathcal{H}$ is separable. However, not every RKHS is separable. Moreover, a complete countable basis is not implied by the Mercer Theorem, since there is no claim that the given sequence of eigenfunctions is complete, and the Hilbert basis formed by the eigenfunctions of the Hilbert-Schmidt operator $R$ need not be countable.

Are Hastie et al tacitly assuming $\mathcal{H}$ is separable? I see no other way around their claim.

1

There are 1 best solutions below

2
On BEST ANSWER

You are right that in general, there is no reason for a RKHS to be separable. However, if the kernel $K$ is continuous and the set $\mathcal X$ over which it is defined is separable, then the corresponding RKHS $\mathcal H $ will also be separable. Here's a proof sketch :

Recall that $\mathcal H$ is the completion of $H_0 :=\text{span}\lbrace K(x,\cdot)\mid x\in\mathcal X\rbrace$. Hence if we prove that $H_0$ is separable, separability of $\mathcal H$ will follow. If we denote by $\{x_n\}_{n=1}^\infty\subset \mathcal X$ a dense subset of $\mathcal X$, it is enough to check that $$H_1:=\bigcup_{n=1}^\infty \left\{\sum_{i=1}^n \alpha_i K(x_i,\cdot)\mid \alpha_1,\ldots,\alpha_n\in\mathbb Q\right\} $$ Is dense in $H_0$, since it is clearly countable. But it is easy to check that any function in $H_0$ can indeed be approximated by elements of $H_1$ when $K$ is continuous : for $H_0\ni f = \sum_{k=1}^m \beta_kK(y_k,\cdot)$, approximate each $y_k$ by an $x_{n_k}$ such that $\|K(y_k,\cdot)- K(x_{n_k},\cdot)\|$ is sufficiently small, and approximate each $\beta_k$ by sufficiently close rationals $\tilde\beta_k$, and the result will follow (I let you fill in the details).

In Hastie's book, they consider $\mathcal X \equiv \mathbb R^p$, which is clearly separable, and all the kernels they work with are continuous, so in their setting $\mathcal H$ is indeed separable.