Let $\Theta$ be a closed connected manifold, $H$ and $F$ Hilbert spaces and $R \colon H \to \mathbb R$ and $\phi \colon \Theta \to F$ smooth.
What is the functional derivative of $$f(u) := R\left(\int_{\Theta} \phi(\theta) \; \text{d}u(\theta)\right) + \lambda u(\Theta),$$ where $u \in M(\Theta; S^{N})$ is real symmetric $N \times N$ matrix-valued measure on $\Theta$? For this integral to well defined, we need a bounded bilinear product $$F \times S^N \to H$$ (see e.g R. Bartle: A general vector integral), which exists, but I don't specify which one.
I can calculate the Frechet derivative of $f$: Let $v \in M(\Theta; S^N)$. We decompose the smooth part of the objective function $f$ as $f = R \circ \Phi$, where \begin{align*} \Phi \colon M(\Theta; S^N) \to H, \qquad v \mapsto \int_{\Theta} \phi(\theta) \; \text{d}{v}(\theta) \end{align*} is a linear map between real vector spaces. Hence $D \Phi(v)[\sigma] = \Phi(\sigma)$ for all $\sigma \in M(\Theta; S^N)$.
Since $R \colon H \to \mathbb R$ is real-valued, there is the following relation between its gradient $\nabla R \colon H \to H^*$ and its Fréchet derivative: $D R(x)[h] = \langle \nabla R(x), h \rangle_{H^* \times H}$ for $x, h \in H$. The linearity of the Frechet derivative and the chain rule applied to $J$ yields \begin{align*} (D f)(v)[\sigma] & = (D R)\big(\Phi(v)\big)\big[ (D\Phi)(v)[\sigma]\big] = \left\langle \nabla R\big(\Phi(v)\big), \Phi(\sigma) \right\rangle_{H^* \times H} \\ & = \left\langle \nabla R\left(\int_{\Theta} \phi(\theta) \; \text{d}v(\theta)\right), \int_{\Theta} \phi(\theta) \; \text{d}\sigma(\theta) \right\rangle_{H^* \times H}. \end{align*}
If I understood correctly, the functional derivative of $J$ with respect to $u$, $\frac{\delta f}{\delta u}$, has to satisfy $$ (D f)(v)[\sigma] = \int_{\Theta} \frac{\delta f}{\delta u}(\theta) \; \text{d}\sigma(\theta). $$ If the measures were real-valued (i.e. $N = 1$), then I could pull out the integral from the second term in the dual pairing $\langle \cdot, \cdot \rangle_{H^* \times H}$ to get $\frac{\delta f}{\delta u}(\theta) = \left\langle \nabla R\left(\int_{\Theta} \phi(\tilde\theta) \; \text{d}v(\tilde\theta)\right), \phi(\theta) \right\rangle_{H^* \times H}$ (as it is done in Lénaïc Chizat: Sparse optimization on measures with over-parameterized gradient descent), but in higher dimension I can not.
Is there a way to still express the functional derivative of $f$, without specifying a product?
Update
Suppose $F = S^N$ and the bilinear product is the usual matrix multiplication, can the question be answered then?