I was watching a machine learning videos from the caltech course CS 156 and they have a slide where they talk about how radial basis functions (RBFs) can be derived from the following variational regularization problem:
$$ H[h] = \sum^{N}_{n=1} (y_n - h(x_n))^2 + \lambda \sum^{\infty}_{k=0} \int^{\infty}_{- \infty} \left( \frac{d^k h }{dx^k} \right)^2 = \sum^{N}_{n=1} Loss(h(x_n), y_n) + \lambda R(h) $$
My question is, what does:
$$\frac{d^k h}{dx^k}$$
rigorously/precisely mean in this context? The issue that I have is that in the video and in the slide it is unclear what space $x$ lies in, so its hard for me to know if they are actually talking about partial derivatives or gradients or what exactly they are trying to say in that equation. If I pretend that $x$ is just a single variable, then its clear that its just all its derivates should be "small" in a 2-norm sense. However, that doesn't make sense to me because the solution to the variational regularization problem $H[f]$ is a function of multiple variables and not only one. The function they claim that can be derives is the following:
$$ f(x) = \sum^{N}_{n=1} w_n exp( - \gamma \| x - \mu_k \|^2 ) = \sum^{N}_{n=1} w_n e^{ - \gamma \| x - \mu_k \|^2 } $$
Which using the norm notation $ \| \cdot \| $ clearly implies that the argument could lie in whatever arbitrary (infinite or finite) dimensional space. So writing
$$\frac{d^k h}{dx^k}$$
confuses me because its not clear to me if they meant to write a derivative with respect to a vector, or a derivative with respect to all the components of the vector, or if it was meant to be the gradient or the absolute size of the gradient etc. Does someone know what they might have meant? For convenience I will provide the slide from the lecture:
For convenience, he start talking about RBFs and regularization at 1:03:43 or click the link: