Motivated by functional gradient in the paper Functional Gradient Techniques for Combining Hypotheses about Machine Learning. I tried to make it rigorously. Could you please verify if my understanding is correct?
Consider
$X$ is a vector space and $A = {\mathbb R}^X$.
$x_1, \ldots, x_n \in X$ and $y_1, \ldots, y_n \in \mathbb R$ fixed.
an equivalence relation on $A$ by $f \sim g \iff f(x_i)=g(x_i)$ for all $i = \overline{1,n}$.
$B = A/\sim$ and an inner product on $B$ by $\langle f,g \rangle = \sum_{i=1}^n f(x_i) g(x_i)$. This inner product is well-defined since all functions in an equivalence class agree on $\{x_1, \ldots, x_n\}$. Because $\mathbb R$ is a Banach space, $B$ together with $\langle \cdot, \cdot \rangle$ is a Hilbert space.
a loss function $\mathcal L: B \to \mathbb R$.
the map (or more precisely, an equivalence class) $\nabla \mathcal L(f) \in B$ by, for all $x \in X$, $$\nabla \mathcal L(f)(x) := \begin{cases} D_{\mathbf{1}_x} \mathcal L (f) & \text{if } x = x_i \text{ for } i = \overline{1,n} \\0 & \text{otherwise} \end{cases}$$ where $\mathbf{1}_x$ is the indicator function of $x$ and $D_{\mathbf{1}_x} \mathcal L (f)$ is the directional derivative of $\mathcal L$ at $f$ along $\mathbf{1}_x$.
Then I would like to verify that $\nabla \mathcal L(f)$ acts like the gradient in the case of finite dimensional Hilbert space, i.e., $\partial \mathcal L(f) (g) = \langle \nabla \mathcal L(f), g \rangle$ for all $f,g \in B$. We have $$\begin{aligned} \partial \mathcal L(f) (g) &= \partial \mathcal L(f) \left (\sum_{i=1}^n g(x_i) \mathbf{1}_{x_i} \right) \\ &= \sum_{i=1}^n g(x_i) \partial \mathcal L(f) \left ( \mathbf{1}_{x_i} \right) \\&= \sum_{i=1}^n g(x_i) D_{\mathbf{1}_{x_i}} \mathcal L (f) \\&= \langle \nabla \mathcal L(f), g \rangle. \end{aligned}$$