Comparing prediction error on different RKHSs

18 Views Asked by At

Let $\Omega \subset \mathbb{R}^d$ and $k_1(\cdot, \cdot), k_2(\cdot, \cdot)$ be two positive definite kernels defined in $\Omega \times \Omega$. Let $\mathcal{H}_1$ and $\mathcal{H}_2$ denote the Reproducing Kernel Hilbert Spaces (RKHS) associated with $k_1$ and $k_2$ respectively. Let $X = \{ x_1, x_2, \dots, x_n \}$ denote a set of $n$ points drawn from $\Omega$. Let $P_1(x)$ and $P_2(x)$ denote the power functions corresponding to the kernels $k_1$ and $k_2$ w.r.t. the set $X$. Given that $\mathcal{H}_1 \subset \mathcal{H}_2$ can we conclude that $|P_1(x)| \leq |P_2(x)|$?

The above is a generalized statement of what I am trying to show. For simplicity, let $\Omega$ be the unit ball centred at the origin. Consider the case where $k_1(x,y) = k_2(x/2, y/2)$ and $k_2$ belongs to the Square Exponential or Matern family of kernels. Note that the choice of $k_1, k_2$ satisfies the assumption of inclusion of the RKHSs. For such a case, intuitively $|P_1(x)| \leq |P_2(x)|$. This is because under $k_1$, the points are more correlated and hence can predict better. For the general case, my intuition is that since $\mathcal{H}_1$ is smaller function space, one would get a smaller prediction error than $\mathcal{H}_2$. However, I am unable to actually show these intuitive results.

I would really appreciate a proof or any reference/leads for any of the two cases. If it helps, we can also assume that $k_1$ and $k_2$ belong to the Matern/Square Exponential family.