Let $A$ be some arbitrary finite set. (e.g. $\{a,b\}$).
Now let $$S=(A\times B)^n$$ Where $B$ is the set of all possible probability functions on $S$:
$$B=\left\{p\in S\to\mathbb R^+:\int_S p(s)ds=1\right\}$$
This definition of $S$ has a kind of "self-referential" structure. Things tend to go wrong when we introduce self-reference in mathematics, so my question is: Is this a well-defined set? If it is not a set, is it even a well-defined class? Is there any way to make sense of this (e.g. using measure theory), or is it nonsensical?
Note that this definition is inspired by Aumann's concept of an interactive belief system, where $n$ agents have a probability distribution over the set $A$, but also over each other's probability distributions over this set, and over THESE probability distributions, ad infinitum.
This is not a legitimate definition of a set. What you have written down is a sort of "set equation;" there are three possible behaviors of such an object:
There are multiple solutions. "$X=X$" is a trivial example; "$X=X\cup\{\mathcal{P}(x): x\in X\}$" is a slightly less trivial example (proving that there is an $X$ satisfying it requires the axiom (scheme) of Replacement). In this case there may still be a "distinguished" solution - e.g. the least fixed point of a positive inductive definition - but the equation cannot be said to define a set without further elaboration.
There are no solutions. The most obvious example is "$X=\{x: x\not\in X\}$." In this case there's nothing to be done. At least in the usual axioms of set theory, your proposed definition is of this type, the reason being that elements of the transitive closure of $S$ would have higher "rank" than $S$ itself (basically, this is a more general version of the problem of sets containing themselves being ruled out by the axiom of regularity; it breaks into two pieces, showing that every set has a rank and showing that rank is "$\in$-increasing").
There is a unqiue solution. This is the case in which it's at least somewhat reasonable to say that the equation "defines" a set. However, I would still consider this incorrect: rather, if $E$ is such an equation then "the unique solution to $E$" is a definition of a set (and it still requires a proof to be legitimate: we need to show that indeed $E$ has a unique solution). As an example of a nontrivial such equation, consider $$X=\{x: x\in X\iff x\not\in X\}.$$ It's a good exercise to check that this equation has exactly one solution, namely $X=\emptyset$.
See also here.