I wonder if there is some existing work discussing measuring the similarity between two kernels, is there any distance defined between two kernels?
A more specific question related to that is, considering for example two squared exponential kernels with automatic relevance determination:
\begin{equation} k_m(x,x')=\sigma_{m}^2\exp\left(-\frac{1}{2}\sum_{j=1}^{q}\left(\frac{x_j-x'_{j}}{l_{mj}}\right)^2\right) \end{equation} And \begin{equation} k_n(x,x')=\sigma_{n}^2\exp\left(-\frac{1}{2}\sum_{j=1}^{q}\left(\frac{x_j-x'_{j}}{l_{nj}}\right)^2\right) \end{equation} Can we have some relationship between the distance of two kernels (if have) and the Euclidean distance between two vectors of parameters of those kernels, something like ($A$ is some constant): \begin{equation} D(k_{m},k_{n})\leq A d(\{\sigma_{m},l_{m1},\dots l_{mq}\},\{\sigma_{n},l_{n1},\dots l_{nq}\}) \end{equation}
Besides, since Reproducing kernel Hilbert space is also highly related to the kernel function, I am also curious about whether there is some distance between two RKHS and whether this distance has some relationship with the distance between two corresponding kernels.
Thanks!
I think you need to clarify exactly what you would like to achieve with such a distance to get good answers. The reason is that distances come a dime a dozen. Here are just three broad approaches:
Metrics on functions Since a kernel is just a function from pairs of input spaces to the real numbers you can use any applicable metric on functions.
Metrics on positive definite matrices There are a bunch of metrics, norms etc. on the space of n-by-n matrices in general and positive definite matrices in particular.
Metrics on parameterized kernels When you focus on families of kernels which are parameterized by hyper-parameters you can apply any metric on the space of parameters to the kernels.