Edit for Clarity: By Riesz, for a continuous evaluation functional $\delta_x$, there exists a $K_x$ such that $\delta_x(f)=\langle K_x,f\rangle=f(x)$, i.e., the inner product of $K_x$ and $f$ evaluates $f(x)$.
Say we use a Gaussian kernel, so that $K_x$ is a Gaussian function (with one parameter fixed). How does an inner product with a Gaussian centered at $x$ and an arbitrary function $f$ in RKHS, result in evaluating $f(x)$?
I have read that the Dirac Delta function has this property, namely:
$\delta_t(f)=\int_a^bf(x)\delta(x-t)=f(t)$
but I cannot wrap my head around how this property could translate to other types of evaluation functionals.
(Apologies for any sloppiness in notation or thinking, I'm self-studying all of this)
Original, Longer Context
I've been studying the construction of Reproducing Kernel Hilbert Spaces. I think I understand most of the basics, but there is a specific point that keeps nagging me. The Riesz Representation Theorem says that the application of a bounded linear functional on a vector (say $f$) can also be represented as the inner product of a vector (say $K_x$) with $f$.
$\delta_x(f)=\langle K_x, f\rangle=f(x)$
Of course, if $f=K_y$, we get:
$\delta_x(K_y)=\langle K_x, K_y\rangle=K_y(x)$
This makes sense. Now I'd like to take a specific example to illustrate my confusion. Let us assume we are using Gaussian kernels. Then $K_x$ represents a Gaussian function with $x$ fixed, let's assume this looks like $K(\bullet,x)$.
Assume that $f=K_y$ is also Gaussian with $y$ fixed, so that $f=K(y, \bullet)$.
But, if we assume the "natural" inner product definition for functions, then it would look like:
$\langle K_x, K_y\rangle=\int K_x(u)K_y(u)du$
which is obviously not going to evaluate to $K_y(x)$ or $K_x(y)$ or $K(x,y)$, because you are integrating the product of two Gaussians (please correct me if I'm wrong here).
Then, my question is, is the "natural" inner product definition of functions not used in an RKHS, and we simply conjure the inner product in our pre-Hilbert space as $\langle K_x, K_y\rangle=K(x,y)$ to make it convenient to get the results we need?
Stated another way, does the inner product definition in an RKHS construction appear as a consequence of something more fundamental, or is it simply there by definition?
Any insights would be welcome. Thanks in advance for your help!