Let $\mathscr{A}$ be a finite alphabet of symbols, equipped with the discrete topology and a probability measure $\mu = (\mu_\alpha)_{\alpha\in \mathscr{A}}$. Then $\mathscr{A}^\infty = \{(p_i)_{i=0}^\infty : p_i \in \mathscr{A}\ \ \forall i\}$ is a compact metrizable space with the product topology, with a metric given by $d(p,q) = |\mathscr{A}|^{-\ell}$ where $\ell = \min\{i : p_i \neq q_i\}$, and a probability space with a measure $\mu$ such that $\mu\big(q : q_i = \alpha\big) = \mu_\alpha$ for any $\alpha \in \mathscr{A}$, $i\geq 0$.
Let $f : \mathscr{A} \to \mathbb{R}$. For $p \in \mathscr{A}^\infty$, I am interested in defining a function $\mathscr{F}(p)$ which is like "the average value $f$ takes over all the symbols of $p$".
To that end, I defined a sequence of functions $\mathscr{F}_n(p) = \frac1n \sum_{i=0}^{n-1} f(p_i)$, which is the average value of $f$ among the first $n$ symbols of $p$. Each $\mathscr{F}_n$ is continuous and therefore measurable, and $\mathscr{F}(p) = \lim_n\mathscr{F}_n(p)$ is the function I am looking for, if it exists.
To complete the construction, I want to show that $\mathscr{F}(p)$ exists for almost all $p$. How do I show this, if it is true?
My first idea was to rewrite $\mathscr{F}_n(p) = \sum_{\alpha\in\mathscr{A}} \frac{\#_\alpha(p,n)}n f(\alpha)$, where $\#_\alpha(p,n)$ counts the occurences of $\alpha$ among the first $n$ symbols of $p$. We can let $n\to\infty$ here, since ${\#_a(p,n)\over n}$ is the probability that a randomly selected symbol out of $(p_i)_{i=0}^{n-1}$ will be equal to $\alpha$, then $\mu_\alpha(p) = \lim_n{\#_a(p,n)\over n}$ is like the probability that any randomly selected symbol of $p$ will be equal to $\alpha$.
If it can be shown that $\mu_\alpha(p)$ exists for almost all $p$, for any $\alpha \in \mathscr{A}$, then that would finish the construction, since $\mathscr{F}(p) = \sum_{\alpha\in\mathscr{A}}\mu_\alpha(p)f(\alpha)$.
Another interpretation: if $\mathscr{A} = \{0,1\}$, then $\mathscr{A}^\infty$ can be identified with the set of all subsets of natural numbers $\mathcal{P}(\mathbb{N})$; a correspondence given by $A \mapsto \hat{A} = (A_i)_{i=0}^\infty$ when $A_i = 1 \iff i \in A$.
Then, if $f : \{0,1\} \to \mathbb{R}$ is given by $f(x) = x$, then $\mathscr{F}(\hat{A})$ is the natural density of a subset $A$. In that sense, $\mathscr{F}$ existing for almost all $\hat{A}$ would imply that "almost all subsets of $\mathbb{N}$ have a natural density".
I found a proof that $\mathscr F(p)$ is defined for almost all $p$ using the strong law of large numbers.
Let $X_n(p) := f(p_n)$, then $X_n : \mathscr A^\infty \to \mathbb R$ is a random variable such that $\mathbb E(X_n) = \sum_\alpha f(\alpha) \mathbb P\big(f(p_n)=\alpha\big) = \sum_\alpha f(\alpha)\mu_\alpha$, therefore the sequence $\{X_n\}$ is iid, and so almost surely
$$\mathscr F_n(p) = \frac1n\sum_{i=0}^{n-1} X_i \to \sum_\alpha f(\alpha)\mu_\alpha$$
And so, $\mathscr F(p) = \lim_n\mathscr F_n(p)$ exists almost everywhere, and is constant. A similar argument shows that $\mu_\alpha(p)$ exists for almost all $p$, and $\mu_\alpha(p) = \mu_\alpha$.
As it turns out, when $\mu$ gets to determine what counts as "almost everywhere", then $\mu$-almost every $p$ induces a distribution over $\mathscr A$ which is exactly equal to $\mu$. Go figure