I was reading through this paper, and I encountered notation that is confusing to me. It's on reinforcement learning / Markov decision processes, but I aim to explain my question in a way that won't require this background knowledge.
On Page 6, they give an equation for "an approximation to the gradient of the expected average reward":
$$\nabla_\beta \eta = \sum\limits_{i, j \in \mathcal{S}} \pi_i \nabla p_{ij} J_\beta (j)$$
All that you need to know in terms of background knowledge is that $\mathcal{S}$ is a set of "states", and the gradient is with respect to a parameter $\theta$ which $p_{ij}$ and $J_\beta(j)$ depends on.
They then say "the gradient approximation $\nabla_\beta \eta$ can be considered as the integration over the state transition space,
$$\nabla_\beta \eta = \int_{(i, j) \in \mathcal{S} \times \mathcal{S}} \pi_i \nabla p_{ij} J_\beta (j) \mathfrak{C} (di \times dj)$$
where $\mathfrak{C}$ is a counting measure, that is, for a countable space $\mathcal{C}$, and a set $A \subset \mathcal{C}$, we have $\mathfrak{C} (A) = \mathrm{card}(A)$ when $A$ is finite, and $\mathfrak{C} (A) = \infty$ otherwise. Here $\mathrm{card}(A)$ is the cardinality of the set $A$".
I believe this as a Lebesgue Integral (or maybe a Riemann-Stieltjes Integral?) since it has the form $\int f(x) \mathrm{d} \mu(x)$, where $\mu(x)$ is a measure. Either way, what does it mean to integrate over a space defined by a cross product, with respect to a counting measure function (i.e. the cardinality $\mathfrak{C} (di \times dj)$---and what does this mean?).
Thanks in advance!
Aaron Hendrickson's comment is correct. The integral is Lebesgue integral with respect to the counting measure over $S \times S$. Specifically, the measure of a finite subset $A \subseteq S \times S$ is defined as the cardinality of $A$.
In terms of the definition, "$\int_{S \times S} f(i,j) \mathfrak{C}(i \times j)$" is defined as $\sum_{i,j \in S} f(i,j)$, so the integral is just a different way of writing the original sum. I haven't read the paper, but maybe there is some advantage in writing it as an integral.