Formula for inner product in a RKHS

Question

Given a kernel function $k$ for which a Reproducing Kernel Hilbert Space (RKHS) $H$ exists, can I write a formula how to compute the inner product of two functions in $H$? I am, of course, aware that because of the reproducing property, I can compute $\left<f, k(x,\cdot)\right>_H = f(x)$ for $f \in H$. But if I take any two arbitrary functions $f,g \in H$, how would the formula for computing $\left<f,g\right>_H$ look like? For example, if $k$ is a Gaussian kernel but also in general case? Can the inner product formula can be always explicitly written? Thank you!

Answer 1

Hope it is not too late.. But maybe here are some intuitions on how to do it. The short answer is "it is difficult in general". Meaning for a given kernel it is not clear at first sight what $\langle f,g\rangle_{\mathcal{H}}$ does (where $\mathcal{H}$ is a RKHS). But with little assumptions we can do things.

A way to tackle this problem is to look at RKHS as a particular subset of square-integrable functions. With little rigor:

Suppose that you are in a space $(\mathcal{X},\mu)$ where $\mu$ is a finite measure. Suppose that $x\rightarrow k(x,x)$ is integrable w.r.t $\mu$ (which is not a "so" strong assumption). This implies that your RKHS $\mathcal{H}\subset L_2(\mu)$ meaning that your functions in the RKHS are square integrable. Indeed: \begin{equation} \begin{split} \int |f(x)|^{2}d \mu(x)&=\int |\langle f,k(x,\cdot)\rangle_{H}|^{2}d \mu(x)\leq \int \|f\|^{2}_{H} \|k(x,\cdot)\|^{2}_{H}d \mu(x) \\ &\stackrel{*}{=}\|f\|^{2}_{H}\int |k(x,x)|d\mu(x)<+\infty \end{split} \end{equation} where I used that $\|k(x,\cdot)\|^{2}_{H}=|\langle k(x,\cdot),k(x,\cdot)\rangle_{H}|=|k(x,x)|$ due to the properties of kernels.

Consider now the operator $\Sigma$ from $L_2(\mu)$ to $L_2(\mu)$ defined by: \begin{equation} \forall x \in \mathcal{X}, \ [\Sigma f](x)=\int f(y)k(x,y)d\mu(y) \end{equation} This operator is linear and finite. It acts like a kind of averaging on $f$ like a sensor. I am not going into details but it is also positive definite (due to Mercer's theorem) so we can talk about eigenvalues and eigenfunctions. So we can diagonalize it like for matrices and find its square root $\Sigma^{1/2}$. If we moreover assume that $\mathcal{H}$ is dense in $L_2(\mu)$ then $\Sigma^{1/2}$ defines a bijection and more importantly an isometry from $\mathcal{H}$ to $L_2(\mu)$ and we have in particular: \begin{equation} \forall f,g \in \mathcal{H}, \ \langle f,g\rangle_{\mathcal{H}}=\langle \Sigma^{-1/2}f,\Sigma^{-1/2}g\rangle_{L_2(\mu)} \end{equation} This results states that the RKHS $\mathcal{H}$ is the subspace of square integrable functions $f$ such that $\|\Sigma^{-1/2}f\|_{L_2(\mu)}<+\infty$. It controls somehow the degree of smoothness of the functions.

So a way to compute the dot products using the kernel is the following: diagonalize the integral operator and just do a usual dot product of functions. The first part might be tricky but for some kernel and measure $\mu$ we know how to do it.

A good ref for this:

[1] Reproducing Kernel Hilbert Spaces in Probability and Statistics. Alain Berlinet et Christine Thomas-Agnan.